Source-linked AI summary
Multi-Modal Knowledge Graph Construction and Application: A Survey
Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, Nicholas Jing Yuan
TL;DR
Purely symbolic knowledge graphs provide limited grounding in physical-world meaning, motivating multimodal knowledge graphs that connect KG symbols with images. This survey defines MMKGs and systematically reviews their construction and application, analyzing solution strengths, weaknesses, and future opportunities. It concludes that MMKGs provide complementary long-tail, background, and fine-grained knowledge for multimodal tasks, while several visual relation and concept-grounding challenges remain unresolved.
Problem
Purely symbolic knowledge graphs do not adequately ground machine knowledge in physical-world meaning, while systematic review of MMKG research has been lacking.
Method
The paper surveys text-and-image MMKG definitions, construction and application methods, their strengths and weaknesses, and open research opportunities.
Results
MMKGs provide background knowledge for long-tail and unseen entities, interpretable reasoning, additional multimodal features, and explicit fine-grained cross-modal knowledge complementary to VL-PTMs.
Takeaways & Limitations
MMKGs can support multimodal tasks by enriching representations, enabling unseen-object understanding, and supplying explicit cross-modal correlations.
Takeaways & Limitations
Visual relation grounding mainly covers spatial and action relations, while many semantic relations lack sufficient training data and visual grounding remains difficult.
Abstract
from arXiv · showhide
Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions. We finalize this survey with open research problems relevant to MMKGs.
1 INTRODUCTION
The survey motivates MMKGs as a way to ground symbolic knowledge in physical-world modalities and systematically reviews their construction and applications. It emphasizes that MMKGs address limitations of purely symbolic KGs while organizing research challenges, strengths, weaknesses, and opportunities.
- Motivation: Purely symbolic knowledge graphs weaken machines’ ability to describe and understand the real world.The paper motivates grounding symbols such as Dog or Hand-in-waistcoat in non-symbolic experiences.
- Motivation: Grounding symbols in images, sound, and video connects KG knowledge with physical-world referents.This grounding is presented as a way to provide machines with experience-like information about entities and concepts.
- Scope: MMKG construction proceeds from images to symbols through labeling or from symbols to images through grounding.The survey reviews both directions for correlating entities, concepts, relations, and events with images.
- Scope: MMKG applications are divided into In-MMKG tasks addressing KG quality or integration and Out-of-MMKG tasks supporting general multimodal applications.The survey presents how MMKGs are applied in several established multimodal tasks.
- Contributions: The survey claims comprehensive coverage, analysis of solution strengths and weaknesses, and identification of future MMKG opportunities.It focuses on MMKG construction and application for knowledge graphs containing texts and images.
2 DEFINITIONS AND PRELIMINARIES
The paper defines MMKGs by associating KG symbols with non-textual data and distinguishes attribute-based and entity-based representations. It then positions MMKGs as complementary knowledge resources for multimodal learning and downstream tasks.
- Definitions: An MMKG is a KG whose entities, relations, attributes, or values are partially associated with non-textual data such as images, sound, or video.A relation triple can be multi-modalized with an image describing the relation between its subject and object.
- Representations: A-MMKG treats multimodal data as attribute values, while N-MMKG treats multimodal data as entities connected by relation triples.N-MMKGs can therefore represent additional inter-modal and intra-modal relations.
- Representations: N-MMKGs can connect image entities through contain, nearBy, sameAs, and similar relations.Images may also be represented through descriptors such as GHD, HOG, and CLD feature vectors.
- Multimodal Benefits: MMKGs provide background knowledge that enriches representations, especially for long-tail entities and concepts, and supports unseen-object understanding.Symbolic relations can connect seen and unseen visual objects for tasks such as caption generation.
- Multimodal Benefits: MMKGs support interpretable reasoning, fill information gaps with additional modalities, and provide explicit fine-grained cross-modal knowledge complementary to VL-PTMs.The paper highlights advantages in long-tail, background, and fine-grained knowledge.
- Discussion: The survey concludes that multimodal tasks remain limited without large-scale MMKG support and can be further improved when MMKGs are available.It reviews MMKG construction and application alongside multimodal techniques and extraction systems.
3 CONSTRUCTION
MMKG construction associates symbolic KG knowledge with corresponding images through two complementary directions: labeling images with symbols and grounding symbols in images. The survey frames these directions as the main construction solution categories.
- Construction Overview: The essence of MMKG construction is associating entities, concepts, and relations in a traditional KG with corresponding images.The paper identifies two opposite ways to complete this association.
- Image Labeling: Image labeling assigns KG symbols to visual content, using tasks such as component, object, and relational labeling.The figure illustrates segmentation-based components, bounding-box objects, and a PartOf relation between two labeled objects.
- Construction Overview: The two construction categories are image-to-symbol labeling and symbol-to-image grounding.The survey reviews each category separately and compares their differences.
3.1 From Images to Symbols: Labeling Images
The survey reviews image-to-symbol MMKG construction, covering visual entity, relation, and event extraction through supervised, weakly supervised, and vision-language approaches. It identifies annotation scarcity, semantic-scale mismatches, unresolved relation reasoning, and open directions for richer visual knowledge acquisition.
- Overview: Image labeling links visual content to KG symbols through visual entity/concept, relation, and event extraction tasks.These tasks organize image-to-symbol construction according to the type of symbol being linked.
- Visual Entity/Concept Extraction: Fine-grained visual entity extraction remains constrained by the lack of large-scale, well-annotated datasets for concepts and entities.Existing computer-vision datasets are generally coarse-grained and do not meet MMKG annotation requirements.
- Visual Entity/Concept Extraction: Visual entity extraction uses object recognition to classify detected regions or visual grounding to map caption phrases to relevant regions.Weak supervision from image-caption pairs reduces reliance on bounding-box annotations, while attention- and saliency-based methods produce relevance heatmaps.
- Visual Entity/Concept Extraction: Visual grounding achieves 69.2% accuracy on Flickr30k, but inconsistent semantic scales can map one phrase to relevant yet nonequivalent visual objects.Examples include mapping “troops” to several individuals or “Ukraine” to a Ukrainian flag.
- Visual Entity/Concept Extraction: Vision-language pre-trained models offer weakly supervised extraction by visualizing image-region and word mappings through self-attention, while recognizing many popular entities.ViLT illustrates region selection for caption words, and CLIP is cited as recognizing popular entities such as famous people and landmarks with high accuracy.
- Visual Relation Extraction: Visual relation extraction includes fine-grained long-tail relations, but recognizing accepted visual knowledge and automatically summarizing reasoning chains remain open problems.The survey highlights context bias, relation hierarchies, and the distinction between scene descriptions and broadly accepted visual facts.
- Visual Event Extraction: Mining and labeling methods can correct incorrect arguments or add missing ones in manually defined visual event schemas.The survey contrasts expected schema items such as Explosion and Weapon with corpus-discovered items such as Smoke and Police.
- Visual Event Extraction: Sequential event extraction from long videos and extraction of multiple sub-events remain underexplored directions for visual event construction.The survey gives Making Coffee as an example of an event decomposed into ordered procedural steps.
3.2 From Symbols to Images: Symbol Grounding
Symbol grounding links KG symbols to images and related multimodal data, with entity, concept, and relation grounding posing distinct coverage, selection, visualization, and semantic-alignment challenges.
- Entity Grounding: Entity grounding links KG entities to corresponding multimodal data, primarily images, using encyclopedia or search-engine sources.These sources must provide enough high-quality images while filtering noisy or weakly related candidates.
- Entity Grounding: Encyclopedia-based construction is easy to scale initially but has sparse, inaccurate, non-visual, and incomplete image coverage.Wikipedia averages 0.83 images per entity; 79.35% of English articles have no corresponding images, and only 6.7% have at least three.
- Entity Grounding: Search-engine approaches improve coverage but introduce noisy images and require query disambiguation and candidate-image cleaning.The two sources are often combined because encyclopedia coverage and search-engine retrieval quality complement each other.
- Entity Grounding: Entity grounding can distinguish visually similar entities and support fine-grained domain MMKGs, while multi-grounding selects context-specific images for multifaceted entities.The survey uses similar people and visual concepts as examples, and Trump’s different roles to motivate contextual image selection.
- Concept Grounding: Concept grounding seeks representative, discriminative, and diverse images, but non-visualizable concepts may require typical entities and remain subjective.Open questions include selecting several typical entities and deciding whether to abstract common visual features across their images.
- Relation Grounding: Relation grounding often relies on co-occurring visual objects or textual entities, yet predefined textual relations may not appear or match in images.Current work emphasizes spatial and action relations, whereas semantic relations such as IsA, Occupation, Team, and Spouse lack sufficient training data.
3.3 Comparing Two Construction Ways
MMKG construction can begin with multimodal data labeling or symbol grounding; their suitability depends on scenario, efficiency, and quality trade-offs.
- Applicable Scenarios: Multimodal data labeling is preferred when images, audio, or video are first-class data, whereas symbol grounding suits symbol-centered settings with redundant or noisy modalities.Which modality is first-class depends on the knowledge the MMKG is intended to provide.
- Efficiency: Symbol grounding is usually more efficient for constructing an MMKG from scratch because labeling methods require time-consuming entity, concept, and relation extraction.Grounding commonly uses retrieval, while multimodal labeling commonly uses classification and detection.
- Quality: Labeling methods face coarse-grained labels and inappropriate semantic hierarchies, while symbol grounding faces missing or mismatching images, especially for long-tail entities.Search results can mislead grounding when a long-tail entity has no suitable web image.
4 APPLICATION
MMKG applications extend multimodal knowledge to tasks within the graph, beyond the graph, and in domain-specific settings.
- 4 APPLICATION: The survey categorizes MMKG applications into in-KG, out-of-KG, and domain applications, and links these tasks to benchmark datasets and MMKG advantages.The overview is summarized in Table 5 before the application sections discuss each category.
4.1 In-MMKG Applications
In-MMKG applications operate on learned multimodal embeddings for graph completion, classification, and alignment tasks.
- MMKG Embedding: MMKG embedding extends conventional semantic-matching and translational KG models to represent multimodal knowledge.The survey introduces semantic matching models such as RESCAL and translational models such as TransE before reviewing applications.
- Application Tasks: The survey reviews link prediction, triple classification, entity classification, and entity alignment as four in-MMKG applications.These tasks use learned entity, relation, and concept representations within the MMKG.
- Link Prediction: Multimodal entity and relation representations enrich link prediction with visual evidence, such as cues about a person’s age, profession, and designation.Link prediction completes missing entities or relations in triples.
- Link Prediction: IMAGEgraph improves relation and head/tail prediction over the conventional approach and generalizes to unseen images for zero-shot visual-relational queries.It represents relations between unseen images as visual-relational queries for MMKG completion.
- Classification and Alignment: Entity classification assigns entities to semantic categories, while entity alignment matches entities with the same real-world identity across MMKGs.Multimodal alignment methods include joint representations and a Product of Experts framework that outperforms concatenation and ensemble approaches.
4.2 Out-of-MMKG Applications
MMKGs support downstream multimodal applications by supplying structured visual, textual, and relational knowledge. The survey reviews their use in recognition, reasoning, retrieval, generation, and recommendation tasks.
- Recognition and Linking: MMKGs assist multi-modal NER and entity linking by supplying complementary visual features, candidate entities, related labels, and distributed entity representations.For disambiguation, systems compare input images with candidate-entity and neighborhood images or retrieve related words from entity co-occurrences.
- Visual Question Answering: MMKGs support VQA by connecting visual and textual entities with external relations, enabling unified representations and explicit reasoning over heterogeneous graphs.They can provide entity relations, re-weight answers, and combine images, questions, and structured facts in a unified format.
- Visual Question Answering: A VQA model fusing explicit MMKG knowledge with implicit VL-PTM knowledge outperforms pure VL-PTMs, while most MMKG knowledge is non-overlapping with VL-PTM knowledge.The MMKG combines hasPart, isA, commonsense, and visual-object location triples from several sources.
- Retrieval: MMKGs improve image-text matching by expanding visual and semantic concepts through entity relations and by providing scene-graph correlations between frequently co-occurring concepts.Examples include concept pairs such as house-window and tree-leaf.
- Multi-modal Generation: MMKGs improve image tagging and captioning by organizing concepts hierarchically, supporting relational reasoning, and connecting seen objects to unseen objects through semantic relations.For tagging, candidate concepts can include parents, children, parts, wholes, synonyms, hypernyms, hyponyms, and related concepts; relational reasoning produces more accurate and reasonable captions.
- Recommendation: MMKGs enhance multi-modal recommendation by enriching item representations, addressing cold-start settings, and selecting smaller reasoning paths for more explicit and explainable recommendations.Reported results indicate that additional structured textual and visual knowledge improves recommendation quality.
4.3 Domain Applications
Domain applications extend MMKG use beyond general multimodal tasks to specialized retrieval and other applications. Examples include geoscience and academic knowledge graphs.
- Domain Applications: Domain MMKGs support multi-hop retrieval, including geoscience queries about locations and affiliations and academic retrieval connecting papers with implementation-level code information.The survey also notes applications in cross-modal retrieval, dialogue systems, and object detection.
5 OPEN PROBLEMS
Open problems concern grounding complex knowledge, ensuring image quality, and scaling MMKG construction and online use. These challenges span representation, data reliability, and efficiency.
- Multiple Relational Grounding: Multiple relational grounding seeks images that express paths or subgraphs containing several semantically related facts.A family photo can ground a subgraph containing Trump’s wife, daughter, and grandson.
- Quality Problems: MMKG image quality suffers from wrong, missing, or outdated visual facts, including images that mix frequently co-occurring entities.The survey identifies these as special quality issues beyond traditional KG concerns such as accuracy, completeness, consistency, and freshness.
- Quality Problems: Grounding can favor images of famous related entities over the intended entity, as illustrated by retrieving The Dark Forest for The Wandering Earth.The problem arises when closely related entities have unequal prominence.
- Quality Problems: Abstract concepts pose additional grounding difficulties because their visual features are not sufficiently fixed or clear.Arrogance is given as an example of a concept with unfixed visual features.
- Efficiency: 350K CPU hours were required by NEIL to collect 400K visual instances for 2273 objects, highlighting scalability challenges for large-scale MMKG construction.Multimedia processing adds complexity beyond typical KG construction, while online applications also require real-time efficiency.
6 CONCLUSION
The survey presents a comprehensive review of MMKG construction and applications, compares approaches, analyzes their strengths and weaknesses, and identifies future opportunities.
- Conclusion: The survey systematically covers MMKG construction and application, compares mainstream resources and methods, analyzes solution trade-offs, and outlines future directions.Its scope is MMKGs constructed from texts and images.