Source-linked AI summary
Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen
TL;DR
Multimodal KGC needs a model that handles diverse tasks and filters irrelevant visual information. MKGformer uses a unified hybrid transformer with multi-level fusion, and experiments report SOTA performance across link prediction, relation extraction, and named entity recognition. Its scope includes settings where MNER and MRE select the top o salient objects according to text.
Problem
Existing multimodal KGC methods require task- and modality-specific architectures, while visual-language pretraining objectives are less relevant to multimodal KGC.
Method
MKGformer uses a unified hybrid transformer with visual, textual, and multimodal encoders, combining prefix-guided interaction with correlation-aware fusion.
Results
MKGformer substantially outperforms current SOTA models across multimodal link prediction, MRE, and MNER in supervised and low-resource settings.
Takeaways & Limitations
The framework supports multiple multimodal KGC tasks through a common architecture with task-specific heads and multi-level image-text modeling.
Takeaways & Limitations
For MNER and MRE, the framework selects the top o salient objects according to the text.
Abstract
from arXiv · showhide
Multimodal Knowledge Graphs (MKGs), which organize visual-text factual knowledge, have recently been successfully applied to tasks such as information retrieval, question answering, and recommendation system. Since most MKGs are far from complete, extensive knowledge graph completion studies have been proposed focusing on the multimodal entity, relation extraction and link prediction. However, different tasks and modalities require changes to the model architecture, and not all images/objects are relevant to text input, which hinders the applicability to diverse real-world scenarios. In this paper, we propose a hybrid transformer with multi-level fusion to address those issues. Specifically, we leverage a hybrid transformer architecture with unified input-output for diverse multimodal knowledge graph completion tasks. Moreover, we propose multi-level fusion, which integrates visual and text representation via coarse-grained prefix-guided interaction and fine-grained correlation-aware fusion modules. We conduct extensive experiments to validate that our MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER. Code is available in https://github.com/zjunlp/MKGformer.
1 INTRODUCTION
Multimodal KGC supports diverse tasks but faces architecture differences and irrelevant visual content. MKGformer addresses these issues with a unified hybrid transformer and multi-level fusion, achieving strong performance across multimodal KGC tasks.
- Multimodal KGs support tasks including recommendation, information retrieval, and forecasting, but remain incomplete because of emerging entities and relations.
- Different multimodal KGC tasks require separately parameterized architectures and fusion modules, limiting a unified model's applicability.
- Irrelevant or redundant images and objects can introduce noise and adversely affect multimodal KGC.
- MKGformer uses a hybrid transformer with multi-level fusion across visual and textual transformer layers to model multimodal entity features.
- Coarse-grained prefix-guided interaction reduces modal heterogeneity, while fine-grained correlation-aware fusion mitigates irrelevant visual content and supports MRE and MNER through task-specific heads.
- MKGformer substantially outperforms current SOTA models across multimodal link prediction, MRE, and MNER in supervised and low-resource settings.
2 RELATED WORKS
Prior multimodal KGC research develops task-specific fusion and alignment methods, while unified modeling remains needed. The paper motivates its approach partly by the mismatch between existing multimodal pretraining objectives and multimodal KGC.
- Existing link prediction methods encode visual features alongside entity and structural knowledge, including through separate representation learning, concatenation, or autoencoding.
- Prior work also studies irrelevant-image noise and proposes selection or forgetting mechanisms to retain more valuable visual information.
- Multimodal relation extraction and named entity recognition use image-text alignment, regional visual features, and multimodal datasets to improve extraction.
- Different tasks and modalities demand architecture changes, motivating a unified model for diverse multimodal KGC applications.
- Existing visual-language pretraining mainly targets image captioning or visual question answering, whose optimization objectives are less relevant to multimodal KGC.
- The paper treats images as supplementary information for enhancing text and proposes a hybrid transformer with multi-level fusion.
3 OUR APPROACH
The approach presents MKGformer as a general framework for widespread multimodal KGC tasks, with unified multimodal processing and task-specific implementations.
- MKGformer is introduced as a general framework applicable to widespread multimodal KGC tasks.
3.1 Unified Multimodal KGC Framework
The unified framework combines visual and textual transformers with task-specific paradigms for multimodal KGC. It models multimodal entities and applies masked-language-modeling-style prediction to link prediction, relation extraction, and named entity recognition.
- The unified framework adopts ViT and BERT as visual and textual transformers and models entity representations across their last L_M layers.
- Multimodal link prediction predicts a missing tail entity from a head entity, relation, and associated images.
- Link prediction follows a masked-language-modeling procedure that first builds image-text entity representations and then predicts the missing entity.
- Entity modeling extends BERT's word-embedding layer with multimodal entity embeddings and trains the model to predict a masked entity representation.
- The model freezes the pretrained parameters and trains newly added multimodal entity-embedding parameters.
- For MRE, the model predicts relation types from text, images, and an entity pair using the final hybrid-transformer representation.
- For MNER, the model predicts sequence tags from text and corresponding images with a CRF-based task head.
3.2 Hybrid Transformer Architecture
MKGformer uses three stacked encoder modules: separate textual and visual encoders followed by a multimodal encoder that integrates their representations. This hybrid design supports multimodal entity modeling across the final layers of both modality-specific transformers.
- MKGformer stacks a T-Encoder for syntactic and lexical information, a V-Encoder for visual features, and an M-Encoder for integrated image-text representations.The M-Encoder operates within the upper layers of the underlying visual and textual transformers.
- The V-Encoder uses the first L_V layers of ImageNet-1k-pretrained ViT to extract features from patched entity images.Each image is reshaped into 2D patches, projected into embeddings, and concatenated across o images.
- For multimodal NER and RE, the model selects the top o salient objects according to the text.
- The T-Encoder uses the first L_T layers of BERT to calculate textual representations from embedded token sequences.Its layers contain multi-head attention and feed-forward blocks.
- The M-Encoder models multimodal entity features across the last L_M layers of ViT and BERT using multi-level fusion.The layer counts satisfy L_ViT = L_V + L_M and L_BERT = L_T + L_M.
3.3 Insights of M-Encoder
The M-Encoder performs coarse-grained modality interaction in self-attention and fine-grained token-patch fusion in FFN layers. These mechanisms reduce modality heterogeneity and limit noise from irrelevant image elements.
- 3.3.1 Prefix-guided Interaction Module.: Prefix-guided interaction performs multi-head attention over hybrid keys and values at every layer to pre-reduce modality heterogeneity.It is applied in the self-attention component of the M-Encoder.
- 3.3.1 Prefix-guided Interaction Module.: The mechanism down-weights visual self-attention by 1 − λ and reallocates λ attention probability to textual attention, yielding coarse-grained fusion.λ denotes the sum of normalized attention weights assigned to textual key and value vectors.
- 3.3.2 Correlation-aware Fusion Module.: Correlation-aware fusion computes token-wise cross-modal interactions, such as word-patch alignment, using similarity-aware aggregation of visual tokens.For each textual token, the module identifies the closest image patches through the similarity matrix.
- 3.3.2 Correlation-aware Fusion Module.: The aggregated visual representation is inserted into the textual FFN calculation to learn fine-grained image-patch and textual-token alignment.The added projection uses parameters W_3 for aggregated visual hidden states.
- 3.3.2 Correlation-aware Fusion Module.: Fine-grained alignment makes the model more robust to noise from irrelevant entity images.
4 EXPERIMENTS
MKGformer is evaluated across multimodal link prediction, relation extraction, and named entity recognition using public datasets, baselines, and supervised or low-resource settings.
- Tasks and datasets: Experiments cover multimodal link prediction, multimodal relation extraction, and multimodal named entity recognition.The link-prediction datasets are WN18-IMG and FB15K-237-IMG; MNRE and Twitter-2017 support extraction tasks.
- Datasets: The datasets include ten images per entity for both WN18-IMG and FB15K-237-IMG.FB15K-237-IMG is derived from Freebase, while WN18-IMG extends WordNet’s WN18.
- Baselines: The evaluation compares MKGformer with text-based, multimodal, and pretrained vision-language baselines.VisualBERT and ViLBERT provide single-stream and two-stream pretrained vision-language comparisons.
- Evaluation settings: Low-resource experiments average test results over five randomly sampled training sets for each task.Few-shot settings are also specified for multimodal link prediction and multimodal relation extraction.
- Visual inputs: For extraction tasks, local visual objects are extracted with a visual grounding toolkit using top-m salient objects.Whole images are used for multimodal link prediction, whereas MRE and MNER use local objects.
4.2 Overall Performance
MKGformer outperforms competitive methods on multimodal link prediction, relation extraction, and named entity recognition, with especially large gains reported for extraction tasks.
- Multimodal link prediction: 1.4% Hits@1 and 3.7% Hits@10 gains over RSME are reported on FB15K-237-IMG.VisualBERT and ViLBERT perform worse than RSME in this comparison.
- Efficiency: The model is described as relatively efficient and straightforward compared with approaches that iteratively query all entities.This efficiency observation is reported alongside the link-prediction results.
4.3 Low-Resource Evaluation
Low-resource evaluations show that generic pretrained vision-language models do not consistently outperform multimodal specialists, whereas MKGformer remains strongest; ablations support its fusion design.
- Vision-language baselines: VisualBERT and ViLBERT slightly improve over unimodal baselines but perform worse than prior multimodal state-of-the-art methods.The pattern is reported across Figure 3, Table 2, and Table 5.
- Low-resource performance: MKGformer outperforms other baselines in low-resource multimodal link prediction, relation extraction, and named entity recognition.The authors interpret this as more efficient use of multimodal data in low-resource settings.
- Data efficiency: The hybrid transformer’s prefix-guided fusion is associated with reduced modal heterogeneity in low-resource settings.The module fuses features in the attention layer using a form similar to linear interpolation.
- Layer sensitivity: Changing M-Encoder layers yields only 0.3% Hits@10, 0.3% F1, and 0.11% F1 improvements on the reported datasets and tasks.The sensitivity analysis covers FB15k-237-IMG, MNRE, and Twitter-2017.
- Ablation study: Removing either prefix-guided interaction or correlation-aware fusion causes performance decay.The ablation results support the effectiveness of both components.
- Fusion placement: Adding fusion outside the transformer reduces performance across three subtasks compared with MKGformer’s internal fusion.The result supports conducting modal fusion inside the dual-stream transformer.
- Multi-level fusion: The coarse- and fine-grained fusion modules are described as mutually reinforcing through progressive reduction of modality heterogeneity.Reduced heterogeneity is proposed to help the correlation-aware module capture fine-grained information.
4.6 Case Analysis for Image-text Relevance
A multimodal relation-extraction case analysis examines whether MKGformer remains robust when image-text relevance varies.
- Irrelevant image-text pairs: MKGformer alone correctly predicts a relation when the image expresses abstract semantics irrelevant to the text.VisualBERT and MEGA fail in the irrelevant image-text case, while text-based BERT and MKGformer predict correctly.
- Representation analysis: Prefix-guided attention highlights relevant entity-object connections, while similarity maps align entity representations with corresponding object patches.These visualizations are presented as evidence of more fine-grained multimodal representations.
5 CONCLUSION AND FUTURE WORK
MKGformer uses a unified hybrid Transformer with multi-level fusion for multimodal knowledge graph completion across several tasks. The authors also identify broader applications and pre-training as future directions.
- MKGformer uses an M-Encoder with multi-level fusion across the final ViT and BERT layers for image-text entity modeling.
- The unified Transformer architecture supports multimodal link prediction, multimodal relation extraction, and multimodal named entity recognition.
- Prefix-guided interaction reduces modality heterogeneity, while correlation-aware fusion performs token-wise fine-grained fusion to mitigate noise from irrelevant images or objects.
- Future work will extend the approach to additional multimodal language tasks, visual representation enhancement, and multimodal knowledge graph pre-training.