PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Single-Negative Preference: Multi-Negative DPO for LLM-Centric Historical Entity Linking

Tien Nam Nguyen, Emanuela Boros, Ahmed Hamdi, Adam Jatowt, Mickaël Coustaty, Antoine Doucet

arXiv:2609.07379v2cs.CLcs.AI

TL;DR

Historical entity linking is difficult in noisy, multilingual archival text, and common LLM preference training uses only one negative despite retrieving multiple candidates. This paper introduces a multi-negative pairwise DPO approach and reports improvements over supervised and unsupervised baselines across several languages, while noting that retrieval remains a bottleneck.

  • Problem

    Historical entity linking faces OCR noise, spelling variation, obsolete or multilingual forms, incomplete metadata, and NIL mentions, while existing LLM approaches often use fixed candidate sets and preference training may use only one negative.

    Source: §1 · pp. 1–2

  • Method

    The framework uses LLM-guided candidate retrieval and trains a reference-based pairwise DPO objective on all valid gold-versus-negative comparisons for each mention.

    Source: §1 · p. 2

  • Results

    Multi-DPO improves over supervised and unsupervised baselines across multiple languages, performs best on all hipe-2020 subsets and three of four newseye subsets, with newseye-FI the exception.

    Source: §6 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The conclusion identifies candidate ranking as the primary challenge and reports that retrieval quality, context design, and preference optimization contribute to final performance.

    Source: §7 · p. 9

    The approach still struggles with NIL prediction because the LLM may generate plausible entities when none exists in the knowledge base, increasing false positives.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have recently shown promise for historical entity linking, but preference optimization for this task is often formulated with only one negative candidate per training instance. This discards information from the remaining candidates retrieved for the same mention. We introduce multi-negative direct preference optimisation (MDPO), a reference-based pairwise objective that compares the correct entity with all valid rejected candidates associated with each mention. MDPO preserves the Bradley-Terry formulation of DPO while exploiting the complete candidate set through masked, length-normalised sequence scores. We evaluate MDPO on hipe-2020 and newseye, covering French, German, English, Swedish, and Finnish historical newspaper text. Experiments show that MDPO improves over supervised fine-tuning and single-negative DPO, with particularly strong gains for NIL mentions, semantic ambiguity, OCR noise, and historically difficult names. Further analyses disentangle candidate-generation and selection errors, showing that candidate retrieval remains a key bottleneck for end-to-end entity linking. These results demonstrate that incorporating all within-instance negative candidates is a simple and effective improvement for LLM-based historical entity linking.

1 Introduction

Historical entity linking is challenged by noisy, multilingual, and incomplete archival text, while many LLM approaches remain auxiliary and do not directly model multiple candidates. The paper proposes an LLM-centric retrieval-and-selection framework and evaluates multi-negative preference learning on two historical benchmarks.

1 Introduction

  • OCR noise, spelling variation, obsolete or multilingual forms, incomplete metadata, and absent entities make historical entity linking and NIL prediction difficult.The introduction situates hipe-2020 and newseye as multilingual historical newspaper benchmarks.Source: §1 · p. 1
  • Many existing LLM approaches augment context, rerank retrieved candidates, or predict from fixed candidate sets, without directly modeling multi-candidate disambiguation.This limits their ability to generate plausible candidates and account for several competitors for one mention.Source: §1 · pp. 1–2
  • The proposed framework uses LLMs for candidate retrieval and entity selection, combining generated candidates validated against a knowledge base with alias-based retrieval.The authors motivate this combination for historical text with OCR noise, spelling variation, and archaic surface forms.Source: §1 · p. 2
  • Multi-negative DPO retains a reference-based pairwise formulation while optimizing all valid gold-versus-negative comparisons in each retrieved set.This contrasts with standard DPO's single chosen–rejected comparison (Rafailov et al., 2023).Source: §1 · p. 2
  • The evaluation covers hipe-2020 and newseye and compares supervised fine-tuning, standard DPO, prompting baselines, and multi-negative DPO, alongside ablations.Ablations examine retrieval, prompt design, context representation, and model choice.Source: §1 · p. 2

2 Related work

Prior LLM-based entity-linking work uses prompting, retrieval, context augmentation, or hybrid reranking, while preference-learning approaches differ in how they use candidate sets. The paper positions its objective as retaining reference-relative pairwise DPO while applying it to all retrieved competitors for a mention.

2 Related work

  • LLM entity-linking approaches include instruction-tuned models with retrievers such as INSGENEL (Xiao et al., 2023), prompted disambiguation such as EntGPT, and selective LLM reasoning in ARTER (Li et al., 2025).Other work enriches context with Wikipedia-like titles (Vollmers et al., 2025) or plugs generated context into existing linkers through LLMAEL (Xin et al., 2025).Source: §2 · p. 2
  • Historical and domain-specific linking work addresses multilingual data, NIL prediction, low-popularity entities, temporal constraints, OCR noise, and domain adaptation.Examples include BELA (Plekhanov et al., 2023) and MHEL-LLaMo (Santini et al., 2026).Source: §2 · pp. 2–3
  • Unlike LiPO's ranked-list loss or single-negative DPO, this work aggregates independent reference-relative gold-versus-negative terms for an unordered retrieved competitor set.The passage distinguishes this task-specific construction from a new listwise objective.Source: §2 · p. 3

3 Methodology

The methodology combines LLM-guided retrieval with LLM-based entity selection, then trains selection using all valid gold–negative comparisons for each mention. Retrieval candidates are validated and supplemented with aliases; the preference objective remains pairwise rather than listwise.

3 Methodology

  • Entity linking selects an entity from a knowledge base for a mention in context, or abstains with a special NIL label.The pipeline has two stages: LLM-guided candidate retrieval and LLM-based entity selection.Source: §3 · p. 3

3.1 LLM-guided Candidate Retrieval

  • LLM-guided retrieval generates plausible candidate strings from a mention and its context, including canonical names, aliases, normalized spellings, and historical variants.These strings expand knowledge-base retrieval rather than serving as final predictions; lexical retrieval can be brittle under historical surface-form variation.Source: §3.1 · p. 3
  • Generated retrieval candidates are validated against the knowledge base and combined with alias-based matches, with duplicate candidates removed.Validation addresses hallucinated or nonexistent names; the alias dictionary is built from each benchmark's training split and retains one-to-many mappings.Source: §3.1 · pp. 3–4

3.2 LLM-based Entity Selection

  • The selection stage scores candidates jointly from the mention, context, and candidate descriptions, framing preference learning around a gold entity and negative candidates.The task supplies multiple valid comparisons for a mention, rather than requiring sampling just one negative.Source: §3 · p. 3, §3.2 · p. 4
  • MDPO uses length-normalized sequence scores and a validity mask to average independent reference-based pairwise terms over real negatives.The mask excludes padding or unavailable comparisons; each valid comparison receives equal weight, while mentions with more hard negatives contribute more pairwise evidence.Source: §3.2 · pp. 4–5
  • The multi-negative objective has no softmax, negative-to-negative interaction, or shared candidate-set normalization; with one valid negative, it reduces to single-negative DPO.“Multi-negative” describes the within-mention comparisons, not a listwise probability distribution.Source: §3 · p. 3, §3.2 · p. 5

4 Preference Data Construction

Preference data are built automatically from entity-linking annotations and candidate sets, with rules for present or missing non-NIL gold entities and NIL gold labels. Records preserve all valid rejected candidates, while instances without a comparison are excluded from DPO training.

4 Preference Data Construction

  • Each automatically constructed instance contains a mention, context, and gold entity, which may be a knowledge-base entity or NIL.Candidate generation supplies the set used to construct preference comparisons.Source: §4 · p. 5
  • When a non-NIL gold entity is present among candidates, it is preferred over every other candidate; when missing, it is added before constructing those comparisons.Adding the missing gold ensures that the model observes the correct entity during training.Source: §4 · p. 5
  • For NIL gold labels, NIL is preferred over all retrieved candidates, or over all non-NIL entities when NIL is already in the candidate set.Source: §4 · p. 5
  • Each training record groups one prompt, one chosen entity, and the complete list of valid rejected entities; records without a valid rejection are excluded.This includes empty-candidate gold-NIL cases, which supply no preference comparison.Source: §4 · p. 5

5 Experiments

Experiments cover multilingual historical entity linking on hipe-2020 and newseye, using LLMs for candidate generation and selection. The evaluations compare prompting, supervised fine-tuning, standard DPO, and multi-negative DPO, reporting micro-F1 and retrieval recall.

5.1 Datasets

  • hipe-2020 covers German, French, and English historical newspapers, while newseye covers French, German, Finnish, and Swedish entity linking.hipe-2020 contains 17,553 named-entity mentions; newseye contains 30,580 named entities, including 6,704 aligned with Wikidata.Source: §5.1 · p. 5

5.2 Models and Implementation

  • Candidate generation uses zero-shot LLMs, while entity selection fine-tunes Qwen3-14B, GPT-20B, or Qwen3-32B with multi-negative DPO.The design separates candidate generation from ranking to use large models for high-recall retrieval while maintaining downstream selection efficiency.Source: §5.2 · p. 6

5.3 Compared Systems

  • The compared selection strategies are prompting, SFT, standard DPO with one negative, and multi-negative DPO with several negatives per instance.The multi-negative variant contrasts the gold entity against multiple rejected candidates within the same instance.Source: §5.3 · p. 6

5.4 Evaluation Metrics

  • Micro-F1 is the primary evaluation metric, with entity-level retrieval recall reported separately to assess candidate retrieval.Micro-F1 follows the HIPE 2020 shared-task evaluation protocol.Source: §5.4 · p. 6

6 Results

Multi-DPO improves on standard DPO across most evaluated language subsets, while analyses show remaining errors in both candidate retrieval and entity selection. Retrieval performance varies by language and model, and NIL handling remains a notable weakness.

6 Results

  • Multi-DPO achieves the best results on all hipe-2020 subsets and three of four newseye subsets, improving on standard DPO in most languages.MELHISSA remains strongest on newseye-FI; standard DPO also substantially improves performance across both datasets over SFT and prompting in most settings.Source: §6 · p. 7

6.1 Oracle Entity Selection Analysis

  • Oracle gains range from 7.7 to 23.3 points, showing substantial selection headroom even when non-NIL candidate recall is high.The oracle substitutes the gold QID only for non-NIL mentions whose gold QID was retrieved; other predictions remain unchanged.Source: §6.1 · p. 7, Table 2 · p. 7

6.2 Retrieval Analysis

  • Retrieval varies substantially across languages and datasets: Gemma4-31B leads hipe-2020 French and German, while GPT-120B leads English and newseye Swedish and Finnish.On newseye, Qwen3-A30B leads French and German but performs poorly on the Nordic languages.Source: §6.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • Simple prompts, chunk-based context, and alias augmentation yield the best retrieval performance; alias lookup provides the largest improvement.Chunk context outperforms summary context, which can omit useful information; simple prompts generally recover more correct candidates than complex instructions.Source: §6.2 · p. 8, Table 5 · p. 8

6.3 Selection Analysis

  • Larger selection backbones do not necessarily perform better after fine-tuning: Qwen3-32B degrades on several newseye languages, while GPT-20B consistently benefits and performs best overall.The authors suggest noisy-supervision optimization instability for Qwen3-32B and stronger compatibility with preference optimization for GPT-20B.Source: §6.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8

6.4 Error Analysis

  • Non-NIL retrieval recall ranges from 83.4% to 88.5%, but NIL recall ranges from 17.0% to 40.7% because the system often retrieves a plausible entity instead of abstaining.Simple prompts generally recover more correct candidates; complex prompts may reduce empty outputs while producing more incorrect candidates.Source: §6.4 · pp. 8–9, Figure 2 · p. 9, Figure 3 · p. 9
  • Fine-grained selector gains are largest for gold-NIL/knowledge-base coverage (+13.7%) and semantic ambiguity (+8.1%), but low absolute NIL retrieval recall and false NIL predictions remain limitations.These are relative accuracy improvements, not solved cases; the analysis also includes OCR noise and other heuristic error types.Source: §6.4 · p. 9, Table 10 · p. 9

7 Conclusions

The paper combines LLM-based candidate generation with preference-optimised entity selection for historical entity linking. Experiments show consistent improvements over supervised and unsupervised baselines across multiple languages, while retrieval quality, context design, and preference optimisation all contribute to performance.

7 Conclusions

  • The framework combines LLM-based candidate generation with preference-optimised entity selection for historical named entity linking.The authors identify candidate ranking as the primary challenge in multilingual and OCR-degraded historical documents.Source: §7 · p. 9
  • Experiments on hipe-2020 and newseye show consistent improvements over supervised and unsupervised baselines across multiple languages.Source: §7 · p. 9
  • Ablations indicate that retrieval quality, context design, and preference optimisation all contribute to final performance.Moderate context windows and simple prompting strategies are generally the most effective.Source: §7 · p. 9

Limitations

The authors identify limitations in NIL prediction, dependence on candidate retrieval, objective comparisons, and inference cost. They also caution that runtime comparisons across systems are not direct when hardware and software stacks differ.

Limitations

  • The approach struggles with NIL prediction because it may generate plausible entities when none exists in the knowledge base, increasing false positives.This is especially challenging with OCR errors, spelling variation, and incomplete context.Source: Limitations · p. 9
  • End-to-end performance depends on retrieval: if the correct entity is not retrieved or validated, the downstream selector cannot recover it.LLM-guided query expansion may improve recall while introducing noisy or hallucinated candidates.Source: Limitations · pp. 9–10
  • The results do not establish superiority over listwise or contrastive objectives with shared candidate-set normalisation, which were not evaluated.The proposed objective is a decomposable pairwise extension of single-negative DPO.Source: Limitations · p. 10
  • Candidate generation is the dominant inference cost in the measured deployable configuration, and reported latency is not directly comparable across different hardware and software stacks.Source: Limitations · p. 10

A Training and Inference Details

Training retains each mention’s chosen entity and all valid rejected candidates from one retrieval set, comparing the chosen score independently with each negative. The setup also specifies the training configuration, data splits, and inference behavior.

A Training and Inference Details

  • The trainer retains one prompt, one chosen entity, and all rejected candidates from each retrieval set, then averages masked, length-normalised pairwise comparisons against a frozen reference.The reference is the pretrained backbone with LoRA adapters disabled; the implementation uses the objective in Eq. 7 rather than flattening instances into unrelated pairs.Source: A Training and Inference Details · p. 12
  • Training runs for 3 epochs with AdamW 8-bit optimisation at 5 × 10−6 learning rate, batch size 8, and gradient accumulation over 8 steps.The context window is 2048 tokens, prompts are capped at 1024 tokens, answers at 128, and the generated branch at five candidates.Source: A Training and Inference Details · p. 12
  • On one RTX A6000, candidate generation takes 2.35–2.63 s and selection 0.35–0.50 s, with generation accounting for approximately 82–89% of latency.The reported combined time is 2.70–3.13 s; cross-system runtime comparisons are not made because hardware and implementations differ.Source: A Training and Inference Details · p. 12
  • Experiments follow official splits: hipe-2020 French and German train/dev data, English zero-shot evaluation, and newseye French, German, Swedish, and Finnish train/dev data.Source: A Training and Inference Details · p. 12
  • Preference pairs use GPT-20B candidate generation for multi-negative training, while standard DPO pairs the gold entity with a randomly selected negative.Instances without a valid positive-negative comparison are excluded from DPO training.Source: A Training and Inference Details · p. 12

B.1 Error Analysis Across Retrieval and Selection

Selection improves NIL prediction accuracy but can reduce QID accuracy, leaving overall accuracy with only modest changes. The analysis also reports a context-length trade-off and qualitative cases involving fine-tuning and disagreements between static NIL annotations and later Wikidata coverage.

B.1 Error Analysis Across Retrieval and Selection

  • Selection improves NIL prediction accuracy by up to +36% across datasets, while QID accuracy drops by up to -26% and overall accuracy changes only modestly.The comparison is entity-level accuracy before and after selection.Source: Figure 4 · p. 13, §B.1 · p. 13

B.2 Context length.

  • A 256-token context window yields the best overall performance; shorter contexts lack disambiguating information, while 512-token contexts can add noise.The authors describe a trade-off between contextual coverage and irrelevant information.Source: §B.2 · p. 13, Table 11 · p. 13

D.1 Success Cases and Comparison with Baseline LLM

D.2 Wikidata Annotation Discrepancies (Gold NIL vs. Predicted QID)

  • Static benchmark NIL labels can disagree with later Wikidata coverage: the system sometimes retrieves plausible, contextually appropriate entities for NIL-annotated mentions.The paper presents these as historically consistent predictions despite disagreement with the original annotation.Source: §D.2 · p. 15

D.3 Key Qualitative Insights

Finetuning corrects a historical-name error involving duplicate or inactive Wikidata entries, distinguishes municipalities from larger regions, and links some mentions labeled NIL to active entities.

D.3 Key Qualitative Insights

  • Finetuning resolves Bartenstein to Bartoszyce (Q809585), where the baseline selects an incorrect duplicate or inactive Wikidata entry (Q6467766).Bartenstein refers to the historical East Prussian city now known as Bartoszyce.Source: §D.3 · p. 15
  • Baseline LLMs sometimes select broader geographic entities, while the finetuned model more reliably distinguishes municipalities using local context.Examples include Berne linked to the canton rather than the city, and New-York linked to the state rather than New York City.Source: §D.3 · pp. 15–16
  • The model links some mentions labeled NIL to active Wikidata entities, including Marie Pospischil and M. Rault; the paper presents these as annotation discrepancies.Table 16 concerns mentions labeled NIL in the official gold standard that the finetuned model links to active Wikidata QIDs.Source: §D.3 · p. 16, Table 16 · p. 16
Loading 2609.07379v2…