Source-linked AI summary
More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning
Chenyi Xiong, Yan Zhang, Jing Hu, Ziyue Qin, Kui Xiao, Xiaopan Lyu, Xiaoju Hou, Zhifei Li
TL;DR
MMKGC needs representations that retain fine-grained modality cues while filtering unreliable heterogeneous signals. PrismF combines multi-perspective enhancement with progressive, reliability-aware fusion, achieving the strongest overall performance across three benchmarks, including gains on sparse KVC16K.
Problem
Existing MMKGC methods may lose fine-grained discriminative cues within modalities and inadequately calibrate noisy, ambiguous, or incomplete cross-modal signals.
Method
PrismF decomposes each modality into complementary perspectives and progressively calibrates modality contributions during fusion using enriched representations.
Results
PrismF achieves the strongest overall performance across DB15K, MKG-Y, and KVC16K, with 4.04% relative improvement in MRR on KVC16K.
Takeaways & Limitations
Effective multimodal entity representation learning requires preserving discriminative intra-modal information and reliability-aware calibration of cross-modal evidence.
Abstract
from arXiv · showhide
Learning effective multimodal entity representations is fundamental for reasoning tasks such as multimodal knowledge graph completion (MMKGC). However, existing methods often suffer from semantic over-smoothing within modalities and ineffective noise filtration across modalities, particularly under sparse or ambiguous conditions. To overcome these limitations, we propose PrismF, a unified framework that synergizes multi-perspective enhancement with progressive fusion to extract stronger signals from diverse inputs. PrismF enhances fine-grained intra-modal semantics through a multi-perspective mechanism that decomposes each modality into complementary views and constrains them with a decoupling loss to reduce representation collapse. Furthermore, it improves cross-modal integration through a progressive fusion strategy that dynamically calibrates inter-modal interactions, enabling the model to emphasize informative signals while suppressing noisy or unreliable ones. Extensive experiments on three public benchmarks show that PrismF achieves the strongest overall performance, including relative improvements of 4.04% in MRR and 11.17% in Hits@1 on KVC16K. Our code can be found at https://github.com/HubuKG/PrismF.
1 Introduction
MMKGC must preserve fine-grained cues within modalities while calibrating heterogeneous signals according to contextual reliability. PrismF addresses these coupled challenges with multi-perspective enhancement and progressive modality fusion, and experiments show consistent gains across benchmarks.
- MMKGC is needed because incomplete and sparsely connected multimodal knowledge graphs hinder reliable inference.
- Existing encode-and-fuse methods extract and combine modality features but may fail to preserve discriminative cues or account for contextual reliability.
- Textual ambiguity and visual similarity can homogenize entity representations, while noisy or incomplete modalities can trigger reliance on a dominant unreliable signal.
- PrismF expands each modality into complementary perspectives with a decoupling objective, then progressively calibrates modality contributions during fusion.
- PrismF consistently outperforms strong baselines across multiple metrics on DB15K, MKG-Y, and KVC16K.
2 Related Work
Prior MMKGC work incorporates textual and visual signals through alignment and adaptive fusion, but PrismF couples intra-modal perspective enrichment with inter-modal reliability calibration. This design preserves modality-specific evidence before fusion rather than focusing primarily on final-stage integration.
- 2.1 Knowledge Graph Completion: Traditional KGC methods mainly model structural information, motivating MMKGC methods that incorporate textual and visual entity signals.
- 2.2 Multimodal Knowledge Graph Completion: MMKGC research explores refined cross-modal alignment, including optimal transport, alongside dynamic and adaptive fusion strategies.
- 2.2 Multimodal Knowledge Graph Completion: PrismF differs by coupling intra-modal perspective enrichment with inter-modal reliability calibration before final multimodal integration.
- 2.2 Multimodal Knowledge Graph Completion: The framework comprises Multi-Perspective Enhancement, Progressive Modality Fusion, and Link Prediction as successive components.
3 Methodology
PrismF progressively enhances modality-specific representations and fuses them using query- and relation-aware calibration. Its components preserve complementary fine-grained cues, reduce redundancy, and dynamically weight heterogeneous signals.
- Framework overview: PrismF comprises Multi-Perspective Enhancement, Progressive Modality Fusion, and Link Prediction for progressively improving multimodal entity reasoning.The framework enhances modality-specific representations, integrates multimodal information, and performs joint prediction in multimodal knowledge graphs.
- Multi-Perspective Enhancement: MuPE decomposes each modality into multiple learnable perspectives to capture fine-grained intra-modal features beyond single-view embeddings.It uses modality-specific inputs, including pretrained image and text features and structurally learned features, to construct perspective-specific representations.
- Multi-Perspective Enhancement: MuPE aggregates perspectives according to relation-specific importance, selectively emphasizing informative views while maintaining diversity and robustness.Relation-aware normalization calibrates perspective weights for different query relations.
- Multi-Perspective Enhancement: A decoupling objective encourages distinct perspectives by reducing off-diagonal Gram-matrix correlations while keeping diagonal values near one.This promotes complementary information without inflating vector norms.
- Progressive Modality Fusion: PMF dynamically recalibrates modality contributions using query-specific confidence scores, training-progress gating, mutual visual-textual gating, and relation-aware signals.The design addresses modality dominance and collapse by combining modality-specific representations according to contextual reliability.
- Cross-modal alignment: For structural features, PrismF applies a relation-aware projection to align stable graph-level signals with the query relation.This improves the structural modality’s alignment and discriminative utility during relational reasoning.
4 Experiments
PrismF is evaluated through link prediction on three public MMKGC benchmarks using rank-based metrics and comparisons with 17 baseline models.
- The experiments address baseline comparison, modality effects, component contributions, hyperparameters, robustness, and qualitative effectiveness.
- PrismF is evaluated on DB15K, MKG-Y, and KVC16K, each containing structural, visual, and textual modalities.
- Performance is measured on link prediction using MRR and Hits@K metrics for K=1, 3, and 10.
- The comparison includes 17 unimodal and multimodal KGC baselines.
- The study reports a controlled implementation using Adam, tuned hyperparameters, and identical experimental settings for all baselines.
4.2 Performance Comparison (RQ1)
PrismF outperforms the compared baselines across metrics, with especially strong top-rank gains and stable improvements on sparse, modality-imbalanced data.
- PrismF achieves the best performance across all evaluated metrics.
- 8.98% relative Hits@1 improvement occurs on DB15K, compared with 1.80% on MKG-Y and 11.17% on KVC16K.
- 4.04% MRR, 6.14% Hits@3, and 3.26% Hits@10 gains are reported on KVC16K.
- The reported results attribute PrismF’s stronger reasoning performance to preserving fine-grained evidence while calibrating heterogeneous signals by contextual reliability.
4.3 Effects of Modality (RQ2)
Ablation and modality analyses show that structural, visual, textual, and fused representations contribute complementary information, with fusion especially important under uncertainty.
- Removing any single modality consistently degrades performance, indicating complementary information across structural, visual, and textual inputs.
- Removing the fused multimodal representation causes the largest performance drop among the modality ablations.
- Figure 4 analyzes hyperparameter performance across all datasets.
- Figure 5 compares performance under modality missing and data sparsity scenarios on DB15K.
- The fused representation aggregates modality-specific cues and adaptively emphasizes informative signals under uncertainty.
4.4 Key Components (RQ3)
Ablation results show that every PrismF component contributes to performance, with the most important effects varying across datasets. The results support coordinated use of complementary representations, progressive calibration, and relation-aware weighting.
- All ablated variants underperform complete PrismF, confirming that each component contributes to overall performance.
- 6.59% relative decrease in Hits@1 follows replacing mutual gating with feature concatenation on DB15K, the largest reported DB15K ablation effect.
- 7.23% MRR reduction follows removing PMF on MKG-Y, while removing the relation-dependent temperature term causes a 6.90% decrease.
- 5.48% and 5.24% MRR decreases on KVC16K follow removing MuPE and stochastic refinement, respectively.These results highlight complementary modality-specific representations and controlled perturbations.
- PrismF benefits from coordinated interaction among its components rather than relying on a single module.
4.5 Hyperparameter Analysis (RQ4)
The analysis examines how the scaling factor λ and threshold μ shape the adjustment curve in Equation 10. Appropriate settings balance modality differentiation against stable learning.
- The scaling factor λ controls adjustment-curve sharpness, while threshold μ determines its transition point.
- Small λ produces overly smooth adjustment, whereas excessively large λ causes abrupt changes that may destabilize learning.
- An inappropriate μ can trigger adjustment either too early or too late.
4.6 Complex Scenarios Analysis (RQ5)
PrismF remains effective when modality information is missing or structural supervision is sparse. Its reported results indicate consistent baseline advantages and stronger robustness under reduced evidence.
- Modality Missing: PrismF consistently outperforms all baselines when randomly discarded modality data creates incomplete inputs.Incomplete data significantly degrades MMKGC performance, while PrismF adjusts available modality contributions according to relevance and reliability.
- Data Sparsity: All methods degrade as structural supervision decreases from removing 5% to 25% of training triples, but PrismF declines more gradually.
- Modality Contributions: Figure 6 visualizes structural, visual, textual, and fused modality contributions across all datasets.
- Case Examples: Figure 7 case examples show PrismF assigning larger weights to more informative modalities based on content reliability.The stated purpose is to mitigate noise and ambiguity.
4.7 Qualitative Analysis (RQ6)
PrismF’s fusion analysis shows distinct contributions from structural, visual, textual, and fused representations without persistent dominance by one modality. Case studies illustrate reliability-aware weighting when modalities conflict.
- PrismF maintains distinct structural, visual, textual, and fused contributions across datasets without persistent single-modality dominance.The substantial fused-representation contribution indicates that complementary cross-modal evidence is retained.
- The progressive factor limits premature reliance on concentrated confidence scores during early training.
- Relation-dependent temperature regulates modality-weight concentration across different query relations.
- For the Thunderstruck query, PrismF weights textual evidence higher than irrelevant visual input containing LeBron James, producing the correct prediction.
5 Conclusion
PrismF achieves strong multimodal entity representations by preserving modality-specific discriminative cues and calibrating cross-modal interactions according to contextual reliability. Across three benchmarks, it remains effective under modality-missing and structurally sparse settings.
- PrismF achieves the strongest overall performance across DB15K, MKG-Y, and KVC16K.The framework also remains effective under modality-missing and structurally sparse settings.
- The framework jointly preserves fine-grained modality-specific cues and calibrates cross-modal interactions according to contextual reliability.
- The findings indicate that effective multimodal representation learning requires both heterogeneous-modality integration and reliability-aware preservation of discriminative intra-modal information.
- Future work will explore more general multimodal architectures and foundation-model integration for knowledge-aware multimodal reasoning.