PaperlayerView exact revision on arXiv

Source-linked AI summary

Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages

Michele Ciletti

arXiv:2609.00998v1cs.CLcs.AI

TL;DR

Historical and low-resource WSD lacks reliable source-language inventories and mappings. INSPICIO addresses this gap with an open-vocabulary, LLM-driven pipeline that links contextual tokens to Open English WordNet synsets; its best perception-verb configuration reaches 96% Recall@50, while component ablations show measurable contributions.

  • Problem

    Historical and low-resource languages often lack comprehensive sense inventories and word-to-sense mappings, limiting standard WSD formulations.

  • Method

    INSPICIO uses LLM-generated translations, definitions, and English lemmas to drive hybrid dense–sparse retrieval and diversification over Open English WordNet synsets.

  • Results

    96% Recall@50 is achieved by the best DeepSeek V4 Pro and KaLM-Embedding-Gemma3-12B configuration on the perception-verb test set.

  • Takeaways & Limitations

    The pipeline provides a fully zero-shot, language-agnostic route to retrieving candidate senses for historical and under-resourced languages.

  • Takeaways & Limitations

    The pipeline is stochastic because translation and gloss generation rely on consecutive LLM predictions, allowing errors to propagate between stages.

Abstract

from arXiv · show

Word Sense Disambiguation has advanced rapidly for English and a handful of well-resourced modern languages, but it continues to assume the existence of a sense inventory and a word-to-sense mapping in the source language (Navigli, 2026). These assumptions break down for most historical and low-resource languages, whose dedicated WordNets are either incomplete or still under construction. We present Inspicio, an open-vocabulary retrieval pipeline that links tokens in context to synsets of the Open English WordNet (McCrae et al., 2020) without requiring any source-language inventory or mapping. For each occurrence, an instruction-tuned LLM produces two English translations of the surrounding sentence, a small set of candidate dictionary-style definitions, and a few candidate English lemmas. These outputs drive a hybrid retrieval step that combines dense definition-synset similarity, sparse lemma matching, and Maximal Marginal Relevance re-ranking. We evaluate the pipeline across a 6x6 grid of LLMs and sentence-embedding models on a new bilingual set of manually annotated Latin and Ancient Greek perception verbs, on a subset of PREMOVE dataset (Farina, 2025), and on a diachronic sample of Italian. The best configuration reaches 96% Recall@50 on the perception-verb test set, with each component contributing measurable gains, and remains competitive in the out-of-domain and cross-lingual settings.

1 Introduction

INSPICIO addresses the lack of source-language sense inventories and mappings in historical and low-resource WSD through open-vocabulary retrieval against the Open English WordNet. It uses LLM-generated English hypotheses and hybrid retrieval, then evaluates the pipeline across languages, datasets, models, and retrieval components.

  • Historical and low-resource languages often lack comprehensive WordNets, while manually building or automatically transferring them is costly and can introduce coverage gaps and noise.
  • INSPICIO retrieves Open English WordNet synsets without requiring a source-language sense inventory or word-to-sense mapping.
  • The pipeline combines LLM-generated translations, candidate definitions, and English lemmas with dense similarity, sparse lemma lookup, and Maximal Marginal Relevance.
  • INSPICIO uses English as a pivot and operates in a fully zero-shot, language-agnostic regime requiring no source-language sense-annotated data.
  • The evaluation covers 150 manually annotated Latin and Ancient Greek perception-verb tokens, PREMOVE, and diachronic Italian.
  • The research questions examine retrieval quality, LLM–embedding interactions, dense versus sparse contributions, and generalization across datasets and languages.

2 Background and Related Work

Related work shows that historical-language WSD is constrained by sparse data, fine-grained inventories, and limited transfer, while open-vocabulary approaches use generated definitions instead of closed candidate lists. INSPICIO extends this direction by linking inventory-free source-language hypotheses to the Open English WordNet.

  • Word-Sense Disambiguation for Historical Languages: Historical-language WSD is constrained by sparse annotated data, fine-grained dictionary inventories, and limited cross-lingual transferability.
  • Word-Sense Disambiguation for Historical Languages: Prior Latin work ranges from binary classification and larger dictionary-based sense samples to propagated annotations and comparisons of zero-shot and fine-tuned generative LLMs.
  • Word-Sense Disambiguation for Historical Languages: Ancient Greek research remains limited, with corpus-based, transformer-based, and semi-automatic LLM-assisted approaches addressing selected resources and lexical classes.
  • Open-Vocabulary Word-Sense Disambiguation: Classical WSD commonly depends on source-language mappings, and performance can degrade sharply when those mappings are incomplete.
  • Open-Vocabulary Word-Sense Disambiguation: Open-vocabulary approaches generate free-form definitions rather than selecting from fixed candidate lists, and INSPICIO aligns these hypotheses with the Open English WordNet.

3 Methodology

INSPICIO processes each token through zero-shot English translation and LLM-generated definitions and lemmas, then retrieves and re-ranks candidate OEWN synsets using dense, sparse, and diversity-aware methods.

  • Zero-shot generation: Each token is converted into two English translations, contextual dictionary-style definitions, and candidate English lemmas through zero-shot LLM calls.The translations are literal and natural; the second call generates one to three definitions and one to five lemmas or short expressions.
  • Dense retrieval: OEWN synsets are indexed as documents combining lemmas, glosses, examples, hypernyms, and lexnames for embedding-based retrieval.This representation outperformed gloss-only and lemma-only variants in preliminary tests.
  • Dense retrieval: Each generated definition retrieves its top 50 synsets, whose dense contributions are aggregated with rank-decay weights of 1.0, 0.75, and 0.5.The weighted sum emphasizes the LLM’s highest-ranked definition while reusing the retrieval lists for efficiency.
  • Sparse retrieval: Candidate lemmas query an inverted OEWN index, adding lexically anchored synsets that dense similarity may miss, especially for idiomatic counterparts.The sparse pool is merged with the dense pool before final ranking.
  • Merging and scoring: The merged pool boosts synsets supported by both retrieval modes, assigns lemma-only candidates a fallback score of 0.65, and retains the top 500.The joint-support boost is γ = 0.8, while β = 0.65 preserves lemma-only candidates below jointly supported ones.
  • Diversification: Maximal Marginal Relevance diversifies the ranking with λ = 0.8 and truncates the final list to K = 50.This reduces clusters of near-identical senses while preserving a relevance-oriented ranking.
  • Inspectability: The pipeline records translations, generated artifacts, scores, lemma matches, and definition contributions for inspection and downstream lexicographic reuse.Curators can trace which definition or lemma drove a result and revise prompts, glosses, or annotations.

4 Data

The evaluation data combine a balanced bilingual perception-verb sample with broader diachronic Latin and Ancient Greek motion verbs and a cross-lingual Italian extension.

  • Dataset motivation: The study addresses limited WordNet-linked annotation by using PREMOVE and a newly constructed bilingual perception-verb dataset.PREMOVE contains around 2,800 manually annotated Latin and Ancient Greek preverbed motion-verb tokens.
  • Perception verbs: The new dataset contains 150 manually annotated tokens: 72 Latin and 78 Ancient Greek instances sampled from a corpus balanced across periods, genres, and authors.The sample spans Archaic to Late Greek and Early to Post-Classical Latin.
  • Perception verbs: Perception verbs were selected because their polysemy combines concrete sensory meanings with metaphorical and mental-state extensions.The examples include literal ‘see’ readings and inferential ‘understand, realise’ readings.
  • Annotation: Each occurrence received an OEWN synset from two trained linguists, with disagreements reviewed to produce one gold synset per token.The annotation used the full sentence and surrounding passage when useful.
  • PREMOVE: PREMOVE tests generalization on naturalistically distributed, diachronic preverbed motion verbs with Zipfian frequencies and frequent metaphorical readings.Its genre and authorial range is broad, while the domain remains restricted to motion verbs.
  • Diachronic Italian: The Italian extension contains 100 diachronic motion-verb tokens matched where possible to Latin lexical entries but affected by lexicalisation and figurative drift.Italian descendants therefore differ in semantic transparency from the Latin comparison set.
  • Annotation: Italian annotation followed the same two-annotator protocol and achieved Cohen’s κ = 0.914.

5 Experiments and Results

INSPICIO is evaluated across a 6×6 LLM–embedding grid and three datasets, achieving its strongest result on perception verbs while retaining substantial recall on PREMOVE and Italian data.

  • Experimental setup: The experiments pair six instruction-tuned LLMs with six sentence-embedding models under identical prompts and report Recall@1, 10, 20, and 50.Recall@50 is the headline metric.
  • Perception verbs: 96% Recall@50 is achieved by DeepSeek V4 Pro with KaLM-Embedding-Gemma3-12B on the perception-verb test set.KaLM is strongest for five of six LLMs, while DeepSeek V4 Pro is strongest in four of six embedding columns.
  • PREMOVE: 81.65% Recall@50 is obtained on PREMOVE with the best configuration, roughly 15 points below the perception-verb result.The difference accompanies broader semantic spread, Zipfian sense frequencies, and diachronic and metaphorical drift.
  • Diachronic Italian: 91% Recall@50 is reached on the diachronic Italian set using the same configuration.
  • Cross-setting interpretation: The PREMOVE outcome remains informative despite linguistic distance, diachronic shift, lexicalisation, and figurative or grammaticalised readings.The authors conclude that the English-pivot strategy remains effective when the source language is represented in the LLM’s pretraining.
  • Ablations: 92% Recall@50 remains after removing translation, showing the sharpest ablation drop and indicating that paired translations add information beyond gloss generation.This reduced setup requires one LLM call and halves inference cost.
  • Ablations: 94% Recall@50 remains after removing the lemma boost, while disabling definition-rank decay or MMR does not appear to reduce performance.The dense and sparse components together recover synsets that neither reaches alone.

6 Discussion

INSPICIO’s remaining errors usually reflect fine-grained mismatches between semantically appropriate generated definitions and OEWN synsets, while its hybrid signals improve ranking when they align. Results across ablations and evaluation settings motivate a future reranking stage that could support silver-standard resources for languages lacking dedicated WordNets.

  • Qualitative Error Analysis: INSPICIO often produces semantically appropriate definitions that miss the gold synset because OEWN distinguishes fine-grained senses.A representative error occurs when a paraphrastic definition retrieves lexically adjacent synsets rather than the gold synset.
  • Qualitative Error Analysis: Lemma matching complements dense definition similarity: when both signals align, the gold synset surfaces more easily, but definition-only matches favor lexically similar neighbours.
  • Future Perspectives: A second-stage LLM reranker is proposed to select definitive synsets from the high-recall top-K pool while preserving INSPICIO’s open-vocabulary upstream design.
  • Evaluation Overview: Table 1 compares Recall@50 across LLMs in rows and sentence-embedding models in columns, with the best combination highlighted.
  • Evaluation Overview: Figure 2 maps Recall@1, Recall@10, Recall@20, and Recall@50 across the same LLM–embedding grid, using darker shading for higher relative recall.
  • Future Perspectives: With reranking, INSPICIO could generate silver-standard annotations and help bootstrap dedicated WordNets for languages that currently lack them.

Limitations

The evaluation and design have important scope boundaries: experiments focus on verbs, LLM stages introduce stochasticity, and English routing may miss source-language senses.

  • The evaluation focuses on verbs, so additional testing on nouns, adjectives, and adverbs is needed to strengthen generality.The architecture is agnostic to part of speech, but the reported evaluation does not establish performance across other categories.
  • Two consecutive LLM prediction stages make the pipeline stochastic, allowing translation errors to propagate into gloss generation and retrieval.The translation stage’s relatively high sampling temperature introduces run-to-run variability, although intermediate outputs help trace failures.
  • Routing through English can return only the closest available synset when a source-language sense lacks a perfect English counterpart.The compromise reflects OEWN’s extensive coverage but also its contemporary-English lexicalization patterns.

A Prompts

The appendix specifies prompts for translation, semantic candidate generation, and embedding-based synset retrieval, with structured outputs and guidance for preserving context-sensitive meanings.

  • A Prompts: The appendix presents three prompts: translation, definition and lemma generation, and embedding.The embedding prompt is used only with models that support it.
  • A.1 Translation Prompt: The translation prompt requests literal and natural English versions of each sentence.It asks the model to stay close to source wording in one version and sound fluent in the other.
  • A.1 Translation Prompt: The translation prompt emphasizes representing both literal and metaphorical meanings of specific words.This directs translation toward distinctions relevant to sense retrieval rather than sentence fluency alone.
  • A.1 Translation Prompt: Translation outputs must be valid JSON with exactly the specified keys and no commentary or markdown.The format constraint is repeated in the prompt’s output instructions.
  • A.2 Definition and Lemma Generation Prompt: The definition-and-lemma prompt uses the target token, lemma, sentence, and two proposed translations to generate context-sensitive candidates.It requests 1–3 ranked dictionary-style definitions and 1–5 English lemmas or short expressions.
  • A.2 Definition and Lemma Generation Prompt: Generation guidelines require precise sense distinctions, including negation, metaphorical meanings, and the difference between a verb’s meaning and its arguments.Examples distinguish meanings such as mentally perceiving a situation from perceiving by sight, and lexicalized manner from contextual detail.
  • A.2 Definition and Lemma Generation Prompt: The prompt permits multiple definitions only for genuine ambiguity and otherwise requests one precise output.It also instructs models to keep outputs consistent and return only the specified JSON fields.
  • A.3 Embedding Prompt: The embedding prompt instructs retrieval of the WordNet synset that best matches a dictionary definition.This connects generated semantic descriptions to the synset-retrieval stage.
Loading 2609.00998v1…