MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking

Motivation

• Text alone can be insufficient for short, context-dependent, and ambiguous entity mentions, while images can provide crucial disambiguating context.Source: §1 · p. 1
• Multilingual entity linking has largely used text-only approaches despite the additional challenges of non-English languages and constrained resources.Source: §1 · p. 1

Contribution

• MERLIN formulates multilingual entity linking through multimodal data and provides a BBC-title testbed paired with images across five languages.The dataset contains over 7,000 named entity mentions linked to 2,500 Wikidata entities in Hindi, Japanese, Indonesian, Vietnamese, and Tamil.Source: §1 · p. 1

Findings

• Visual and textual information significantly improves disambiguation over methods without visual information, especially for LLMs that are not massively multilingual.Source: §1 · p. 2
• Even the best evaluated approaches remain limited in performance, demonstrating the difficulty of the MERLIN task and dataset.Source: §1 · p. 2

Task definition

• Each mention is characterized by visual context from its associated image and textual context containing the mention and surrounding spans.Source: §2 · p. 2
• MMEL identifies the Wikidata entity corresponding to a mention given its image and surrounding textual context.The output is a set of mention-entity pairs over the knowledge-base entity set.Source: §2 · p. 2

Assumption

• The multilingual setting requires cross-lingual and cross-modal understanding because textual context may appear in different languages.Source: §2 · p. 2
• The evaluation follows an in-KB paradigm in which every mention is assumed to have a valid corresponding entity, excluding out-of-KB predictions.Source: §2 · p. 2

3.1 Language and Dataset Selection

• MERLIN uses BBC articles from M3LS spanning topics such as politics, sports, economy, science, and technology, with 1,000 titles selected per language.Selection also considered budget, annotator availability, and curation time.Source: §3.1 · pp. 2–3
• The dataset selects Hindi, Japanese, Indonesian, Tamil, and Vietnamese to represent linguistic diversity, speaker populations, and annotator availability.The languages span Austronesian, Austroasiatic, Indo-European, Dravidian, and Japonic families and use multiple scripts.Source: §3.1 · pp. 2–3

3.2 Dataset Annotation

• Annotators were recruited through Prolific with language and nationality requirements, and had to achieve at least 60% F1 on a ten-article English pre-screening task.Source: §3.2 · p. 3, §3.1 · p. 3
• Each language’s 1,000 titles were split into ten groups, each independently annotated by three annotators, with majority voting determining final entities.Source: §3.2 · p. 3

3.3 Dataset Analysis

• Entity types are assigned by traversing Wikidata’s P31 and P279 hierarchy to a root category such as Person, Country, Location, Organization, Event, or Miscellaneous.Source: §3.3 · p. 3
• Cohen’s Kappa is used across annotator pairs and language splits to measure entity-linking agreement, with most values indicating almost perfect consistency.Source: §3.3 · p. 4

mGENRE

• mGENRE uses an mBART sequence-to-sequence model with constrained decoding to generate valid entity names directly from the article title and mention.Its prefix-trie search restricts outputs to Wikipedia entity names.Source: §4 · p. 4

GEMEL

• GEMEL maps mentions from visual and textual contexts to entities while freezing the vision encoder and LLM and training a feature mapper for cross-modal interaction.The method is trained on English Wikidiverse data and evaluated zero-shot on MERLIN.Source: §4 · p. 4

Language models

• Aya-23 replaces Llama-2 as GEMEL’s text encoder because it is trained on diverse multilingual data and is designed for stronger multilingual performance.Llama-2 is described as having limited multilingual capability, while Aya-23 covers 23 languages but lacks Tamil.Source: §4 · p. 4

Knowledge base

• Both baseline systems use Wikipedia titles as their knowledge bases, but constrained decoding is applied to mGENRE and not to GEMEL.Source: §4 · p. 4

5 Results and Analysis

• mGENRE reaches 78.93% Recall@1 in Indonesian, while Tamil falls to 63.18%; GEMEL-Llama-2 reaches 74.76% in Japanese and GEMEL-Aya-23 reaches 67.53% in Hindi.Aya-23 outperforms Llama-2 in Hindi and Tamil, but both remain below mGENRE there; the models otherwise show similar performance.Source: §5 · pp. 5–6, Table 5 · p. 7
• Removing images significantly reduces Llama-2 performance, whereas Aya-23 shows a much smaller drop without images.The authors relate this difference to the models’ multilingual training coverage and hypothesize that images matter more for models lacking strong multilingual training.Source: §5 · p. 6
• Images resolve cases where text-only systems select generic or related entities, correctly distinguishing Wales’s rugby team, Smriti Irani, Southampton F.C., and Boris Johnson.The text-only alternatives link these mentions to Wales, S. M. Krishna, Southampton, and the United Kingdom prime-minister position, respectively.Source: §5 · p. 7, Figure 4 · p. 7
• Visual inputs provide marginal gains for ambiguous mentions in Hindi and Japanese but significantly help Indonesian and Vietnamese, while Tamil remains difficult even with images.Across languages, ambiguous mentions are consistently harder; Llama-2 reaches only 28% in Japanese and 18.18% in Tamil in the text-only setting.Source: Table 7 · p. 8, §5 · p. 8
• Images significantly improve PER linking, whereas ORG mentions are easier overall and MISC and Event mentions remain consistently less accurate.The latter categories appear to require contextual understanding beyond current multimodal approaches.Source: §5 · p. 8

Q3) How does performance of both models differ across languages?

• GEMEL models generally perform better with visual context across all five languages, with the effect especially important for low-resource settings.The comparison includes mGENRE, GEMEL-Llama-2, GEMEL-Aya-23 8B, and variants without images.Source: Table 5 · p. 7, Q3) How does performance of both… · p. 8
• Translation to English generally improves GEMEL performance, particularly for Tamil and Hindi, but evaluation uses a reduced entity space after filtering translation-affected mentions.The filtering excludes entities whose mentions do not appear in both source and translated text, limiting direct comparison with the original multilingual evaluation.Source: Q3) How does performance of both… · p. 9, Table 8 · p. 9

Multilinguality in Entity Linking

• Multilingual entity linking progressed from cross-lingual linking to multilingual models and language-agnostic knowledge bases, including mGENRE’s autoregressive entity generation.Source: §6 · p. 9

Multimodality in EL

• Multimodal entity linking combines textual, visual, and lexical information, but existing datasets remain constrained by privacy, entity coverage, ambiguity, or domain scope.Source: §6 · pp. 9–10
• Recent multimodal methods address semantic inconsistency, attribute integration, and fine-grained cross-modal alignment through contrastive learning, structured attributes, and graph convolutional networks.Source: §6 · p. 10

Multilingual, Multimodal Datasets

• MERLIN addresses the multilingual-multimodal gap with a dataset supporting entity linking across diverse languages and modalities as a benchmark for future models.Source: §6 · p. 10

• MERLIN introduces a multilingual multimodal entity linking dataset covering five languages and scripts, built from BBC news articles with associated images.Baseline systems include mGENRE, LLaMA-2, and Aya-23B.Source: §7 · p. 10
• Visual cues significantly benefit disambiguation when textual descriptions are ambiguous, while multilingual linking remains inherently difficult for low-resource languages.Source: §7 · p. 10

• Using only BBC news articles limits genre diversity and may reduce model generalizability to broader real-world writing styles and entity distributions.Source: §8 · p. 10
• Despite strong Cohen’s Kappa agreement, annotations may contain inconsistencies from differences in language proficiency, knowledge, skill, and human error.Source: §8 · p. 10
• Baseline systems link to Wikipedia page titles rather than directly to Wikidata QIDs, so their results may not represent pure Wikidata-QID linking performance.Source: §8 · p. 10

• Annotators link entities and mentions in news articles to corresponding entries in existing knowledge bases.Source: Entity Linking Annotation Guidelines · p. 14
• Each annotation instance includes a news article title and an associated image, which annotators may use to resolve ambiguity.Source: Entity Linking Annotation Guidelines · p. 14

Terms and Concepts to Understand the Process Better

• Entity linking maps mentions such as names, locations, and companies to corresponding unique entities in a knowledge base.Source: Terms and Concepts to Understand the… · p. 14
• The taxonomy includes Person, Location, Organization, Country, Events, Works, and Miscellaneous entity types.Source: Terms and Concepts to Understand the… · p. 14

Types of Entities and Examples

• Examples classify iPhone 12 as a work, France and Germany as countries, Shohei Ohtani as a person, Dodgers as an organization, and R21 as miscellaneous.Source: Types of Entities · p. 14

How to Access the website

• Annotators access documents through INCEpTION, select text spans alongside images, and use Wikidata search to identify mentions.Source: How to Access the website · p. 14, §1 · p. 15, §2 · p. 15

3. Complete Annotation

• Annotations require choosing an entity type, while unavailable Wikidata entities are marked “No” and left blank after search attempts.Source: §3 · p. 16
• Annotators repeat the linking procedure for every entity span in a document before moving to the next document.Source: §3 · p. 16

Effectively Searching Wikidata Entities

• Wikidata search is sensitive to exact wording, may return duplicate names, and sometimes requires English queries or external searching to identify the correct page.Source: Effectively Searching Wikidata Entities · pp. 16–17
• Visual context helps resolve hard mentions such as “Amazon,” “PM,” and “India,” while Table 10 reports differing visual-input effects across entity categories and languages.Source: Figure 12 · p. 18, Table 10 · p. 19