PaperlayerView exact revision on arXiv

Source-linked AI summary

KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs

Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio, Giovanna Castellano, Ralf Krestel

arXiv:2610.16502v1cs.CVcs.LG

TL;DR

Multimodal entity linking must disambiguate evidence whose relevant context is implicit, while art’s visual diversity complicates matching from images and text alone. KARAMEL combines a vision-language backbone with heterogeneous graph knowledge, and experiments report consistent gains over strong baselines on WikiMuSA. The authors identify adaptive subgraph extraction and stronger robustness to incomplete or noisy knowledge as future work.

  • Problem

    Vision and text models can miss contextual knowledge distributed across modalities, while artistic abstraction and stylistic variation complicate entity linking from visual features.

    Source: §1 · pp. 1–2

  • Method

    KARAMEL combines a vision-language backbone with an inductive heterogeneous graph neural network to integrate visual, textual, and structured contextual information.

    Source: §1 · p. 2

  • Results

    Experiments on WikiMuSA show consistent gains over strong baselines, and ablation results show a performance drop when moving from the full graph to the ablated graph.

    Source: §5.3.3 · p. 11, §6 · p. 12

  • Takeaways & Limitations

    Graph-driven contextual reasoning captures relational dependencies that are difficult to infer from vision or text alone, particularly for abstract or implicitly referenced entities.

    Source: §6 · p. 12

    The approach relies on manually curated predicates, and future work targets scalability and robustness to incomplete or noisy knowledge.

    Source: §6 · p. 12

Abstract

from arXiv · show

Discovering and disambiguating entities from multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior vision-language models that primarily rely on surface-level multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a vision-language encoder with a heterogeneous graph neural network. This design enables contextual reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based Multimodal Semantic data for Art), a new dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving multimodal observations and heterogeneous background knowledge. Both the dataset and source code are made publicly available.

1 Introduction

KARAMEL combines vision-language encoding with heterogeneous graph context to address entity linking when relevant evidence is implicit and art is visually complex. The introduction presents WikiMuSA as a more challenging discovery setting in which depicted entities may need to be inferred rather than extracted from explicit mentions.

1 Introduction

  • Vision-language models can overlook broader contextual information, while directly embedding large knowledge graphs poses scalability and computational challenges.The introduction notes that aggressive subgraph pruning may also introduce inconsistency or data drift.Source: §1 · p. 2
  • KARAMEL combines a vision-language backbone with an inductive heterogeneous graph neural network and reports improved disambiguation from domain-specific contextual knowledge beyond vision-text inputs alone.The framework integrates visual and textual information with structured relational knowledge.Source: §1 · p. 2
  • WikiMuSA requires inferring depicted entities from artwork and article context without predefined textual mentions, unlike standard MEL settings with explicit mentions.The example links “Arch of Nero” to “ruins” using visual evidence and cues such as “ancient Roman architecture”.Source: §1 · p. 2, Figure 1 · p. 2
  • Art’s abstraction, stylistic variation, small depicted regions, and brushwork differences complicate visual matching and motivate structured context beyond vision-language features alone.Historical, social, and political knowledge in descriptions and knowledge bases can support semantic relationships and interpretation.Source: §1 · p. 2

2 Related Work

Related work spans text-based entity linking, multimodal fusion, and cultural-heritage resources, but existing approaches and datasets leave gaps in contextual reasoning and implicit depiction coverage. The paper introduces a Wikidata subgraph and a multimodal framework to address these gaps.

2 Related Work

  • Earlier entity-linking work progressed from word embeddings and convolutional networks to transformer-based methods that encode mentions and rank candidate entities.Examples include knowledge-enhanced BERT, BLINK, entity-label adapters, and temporal linking with graph data.Source: §2 · p. 3
  • Multimodal entity linking combines visual and textual features, with prior work using cross-attention or multi-grained matching to fuse modalities.Earlier work also found that combining visual and textual information can alleviate polysemy in social media analysis.Source: §2 · p. 3
  • Cultural-heritage MEL remains challenging because contextual relationships can be complex and implicit, while prior art datasets leave fine-grained or coverage gaps.ArtGraph lacks a detailed taxonomy for depicted entities; MELArt assumes depicted entities are explicitly mentioned and does not cover all entities shown.Source: §2 · p. 3
  • WikiMuSA is a curated Wikidata subgraph focused on art relationships, designed to balance domain-relevant information richness with scalability.KARAMEL builds on it to integrate images, text, and heterogeneous graph knowledge for representations of depicted entities.Source: §2 · p. 3

3 WikiMuSA

WikiMuSA represents artworks through visual, textual, and contextual information, with graph relations connecting artworks to metadata and depicted entities. Its collection and processing pipeline produces a multimodal resource for cultural-heritage linking.

3 WikiMuSA

  • WikiMuSA’s schema associates artworks with metadata and links depicted entities through enriched contextual relationships that increase expressiveness and graph connectivity.The dataset includes visual, textual, and contextual information relevant to cultural heritage.Source: Figure 2 · p. 4, §3 · p. 4

3.1 Data Collection and Processing

  • The collection includes Wikidata artworks with a linked Wikipedia article, at least one depicts relation, and an associated image; non-English articles were translated and articles summarized for depiction-focused content.The summarization was applied consistently across models to reduce variance and noise while preserving depiction-relevant content.Source: §3.1 · p. 4

3.2 Graph Structure and Relations

  • Artwork nodes connect to contextual nodes such as authors, genres, movements, galleries, and depicted entities, with curated properties varying by entity category.For example, person properties include occupation, gender, and relatives; place properties include neighboring cities and location hierarchy.Source: §3.2 · pp. 4–5

3.3 Final Dataset

  • WikiMuSA contains 13,482 artworks and 12,177 depicted entities, with artwork image-article pairs and entity visual, textual, and graph-derived contextual representations.The dataset is presented as a testbed for multimodal entity linking and knowledge-aware discovery.Source: §3.3 · p. 5

4 Proposed Method

KARAMEL extends vision–text alignment with structured knowledge-graph context to link multimodal mentions to entities. Its ranker combines visual, textual, cross-modal, and contextual knowledge scores.

4 Proposed Method

  • KARAMEL extends vision–text alignment with a dedicated graph encoder to incorporate structured contextual knowledge when linking under ambiguous and incomplete evidence.Source: §4 · p. 5

4.1 Problem Formulation

  • Each mention and candidate entity has visual and textual information, and every candidate corresponds to a Depicted Object node in a typed heterogeneous graph.The objective selects the ground-truth entity among candidates and is extended to condition on the graph.Source: §4.1 · p. 5

4.2 Model Architecture

  • KARAMEL processes artwork mentions, candidate entities, and the knowledge graph with an image-text cross-encoder, a heterogeneous GNN encoder, and a cross-modal ranker.The cross-encoder uses CLIP pre-trained on LAION-400M, while graph convolutions extract context-aware information from the knowledge graph.Source: §4.2 · p. 5

4.2.1 Feature Encoding

  • Visual encoding yields global and local image features, while textual encoding uses BERT representations; missing entity images are zero-padded.The visual encoder uses a ViT backbone, and the textual encoder extracts a global representation from the [CLS] token.Source: §4.2.1 · pp. 5–6
  • The GNN aggregates neighboring-node information into contextual embeddings for candidate depicted entities.Entity labels and descriptions are concatenated and encoded with pre-trained BERT as initial node features.Source: §4.2.1 · p. 6

4.2.2 Cross-Modal Ranker

  • The ranker averages visual, textual, vision-text, and contextual knowledge scores for each mention–entity pair.Its contextual score aligns the cross-attended mention representation with the entity’s knowledge-graph embedding; the authors say this captures historical, religious, and social factors and may apply beyond art.Source: §4.2.2 · p. 7

5 Experiments

The experiments compare KARAMEL with entity-linking baselines and examine how contextual graph information and node features affect ranking. Across the reported analyses, KARAMEL performs best overall, with differences depending on dataset and evidence type.

5.1 Experimental Setting

  • Evaluation ranked all depicted entities using R@K, MAP, and MRR, with artwork-based 80/10/10 splits and validation and test artworks removed from the graph.The textual input used the full article rather than predefined mention spans.Source: §5 · p. 8, §5.1 · p. 8

5.2 Baselines

  • Baselines covered BERT as text-only, CLIP and ViLT as vision-language models, and MIMIC as a multimodal MEL model.MIMIC integrates multigrained analysis of vision and text.Source: §5.2 · pp. 8–9

5.3.1 Performance Comparison

  • KARAMEL outperforms the baselines across all metrics on WikiMuSA, with MIMIC the closest competitor.The authors report consistent improvements over MIMIC, particularly in MAP, and lower variance across seeds.Source: Table 1 · p. 9, §5.3.1 · p. 9
  • On MELArt, KARAMEL improves first-position retrieval and generally improves MAP and MRR over MIMIC, but recall declines at other thresholds.The authors hypothesize the smaller dataset contributes to the smaller model difference and higher run-to-run standard deviation.Source: Table 2 · p. 9, §5.3.1 · p. 10

5.3.2 Feature-Wise Statistical Analysis

  • KARAMEL significantly improves rankings in several cases, especially when entities are unmentioned or pre-iconographic; entities without artwork links favor MIMIC.Entities linked to artworks show improved rankings, and entities associated with predicates such as part of, subclass of, or instance of tend to rank better.Source: Figure 4 · p. 10, §5.3.2 · pp. 10–11
  • In the Odysseus and Polyphemus example, KARAMEL ranks both ground-truth entities first, while MIMIC places only one in its top four.Both systems retrieve semantically plausible candidates, but KARAMEL’s scores align more closely with Wikidata’s significance annotations.Source: Table 3 · p. 11, §5.3.2 · p. 11

5.3.3 Ablation Study

  • Across node-feature encodings, removing contextual edges consistently lowers performance; on the full graph, hybrid leads MAP and MRR while text-only leads Recall@10.The full graph retains rich heterogeneous relations, whereas the ablated graph retains only the shared instance-of-class link.Source: §5.3.3 · pp. 11–12, Table 4 · p. 12

6 Conclusion

KARAMEL combines vision and language with structured relational knowledge for multimodal entity linking in the arts. Experiments on WikiMuSA show gains over strong baselines, while graph-driven context supports reasoning about abstract or implicit entities.

6 Conclusion

  • Experiments on WikiMuSA show consistent gains over strong baselines for multimodal linking.The conclusion highlights domain-relevant knowledge as beneficial for robust linking.Source: §6 · p. 12
  • KARAMEL combines a vision–language backbone with a heterogeneous graph encoder to use contextual dependencies for entity ranking and disambiguation.The setting includes implicit entities and textual mention spans that are not predefined.Source: §6 · p. 12
  • Graph-driven representations capture relational correlations and dependencies that are difficult to infer from vision or text alone, particularly for abstract or implicit references.The proposed design is presented as a general blueprint for knowledge-aware discovery, although this study focuses on cultural heritage.Source: §6 · p. 12
  • Future work includes adaptive subgraph extraction, graph-neighborhood and cross-modal evidence attribution, and handling temporal, evolving, incomplete, or noisy knowledge graphs.Adaptive extraction could reduce reliance on manually curated predicates and improve scalability; explainability could increase transparency and trustworthiness.Source: §6 · p. 12

A Translation and summarization analysis

The analysis describes how Wikipedia articles were translated when needed and summarized to balance article length and detail. The summaries are generally shorter than the originals and show substantial ROUGE-1 overlap, though ROUGE-2 precision drops, likely with language simplification.

A Translation and summarization analysis

Related papers

Loading 2610.16502v1…