Source-linked AI summary
Visual Pivoting for (Unsupervised) Entity Alignment
Fangyu Liu, Muhao Chen, Dan Roth, Nigel Collier
TL;DR
Entity alignment across heterogeneous knowledge graphs is constrained by scarce seed alignments and weak structural correspondence. EVA combines visual, structural, relational, and attribute information, using visual similarities to build pivots without labels. Across DBP15k and DWY15k, it achieves state-of-the-art results, while images are especially useful for long-tail entities; visual coverage and representation stability remain limitations.
Problem
Entity alignment lacks abundant cross-graph pivots, while sparse graphs often provide weak structural correspondence for matching entities.
Method
EVA fuses image, structure, relation, and attribute information with attention weighting and induces visual pivots for unsupervised iterative alignment.
Results
EVA achieves state-of-the-art performance across the reported entity-alignment settings, while its unsupervised version exceeds 70% accuracy and approaches semi-supervised performance.
Takeaways & Limitations
Images provide particularly useful alignment signals for long-tail entities with impoverished structural contexts.
Takeaways & Limitations
Only 50-70% of studied entities have images, and unstable visual representations can cause misalignment for abstract entities.
Abstract
from arXiv · showhide
This work studies the use of visual semantic representations to align entities in heterogeneous knowledge graphs (KGs). Images are natural components of many existing KGs. By combining visual knowledge with other auxiliary information, we show that the proposed new approach, EVA, creates a holistic entity representation that provides strong signals for cross-graph entity alignment. Besides, previous entity alignment methods require human labelled seed alignment, restricting availability. EVA provides a completely unsupervised solution by leveraging the visual similarity of entities to create an initial seed dictionary (visual pivots). Experiments on benchmark data sets DBP15k and DWY15k show that EVA offers state-of-the-art performance on both monolingual and cross-lingual entity alignment tasks. Furthermore, we discover that images are particularly useful to align long-tail KG entities, which inherently lack the structural contexts necessary for capturing the correspondences.
1 Introduction
Entity alignment must reconcile independently built knowledge graphs despite scarce cross-graph pivots and weak structural correspondence. EVA addresses this gap by combining visual, structural, relational, and attribute information, using visual similarities to induce pivots and achieving strong performance in supervised and unsupervised settings.
- Independently built or language-specific knowledge graphs limit coverage, motivating alignment and synchronization across graphs.
- Scarce cross-graph pivots and weak structural correspondence make entity alignment difficult, especially for entities with impoverished contexts.
- Visual information can provide language- and schema-independent alignment signals because entity images are often available, verified, and disambiguated.
- EVA combines images with structures, relations, and attributes through attention weighting, supporting both seed-label and automatically induced visual-pivot training.
- EVA variants consistently outperform competitive baselines on DBP15k cross-lingual and DWY15k monolingual entity alignment benchmarks.
2 Related Work
Prior work studied entity alignment and multimodal knowledge-graph embeddings, but had not incorporated visual modality into cross-graph entity alignment. This paper positions visual pivoting as an unsupervised solution that avoids alignment labels.
- Earlier entity-alignment research included symbolic, schematic, and embedding-based methods for matching entities across knowledge graphs.
- Multimodal knowledge-graph studies combined visual information with embeddings mainly for intrinsic tasks such as word similarity, link prediction, or knowledge-graph completion.
- The paper identifies its contribution as the first incorporation of visual modality into knowledge-graph entity alignment.
- Visual pivoting extends image-based alignment ideas beyond lexical, sentential, and text-image matching tasks to cross-graph entity alignment.
3 Method
EVA embeds entities using graph structure, images, relations, and attributes, then aligns cross-graph entities through multimodal similarity and iterative training. Its unsupervised extension induces visual pivots from visually similar entities and filters proposals through repeated mutual-nearest-neighbour agreement.
- Multimodal entity representation: EVA jointly represents entities with graph structure and auxiliary images, relations, and attributes in a shared embedding space.Alignment learning then captures correspondence between counterpart entities using an NCA-based objective.
- Structural and visual embeddings: A GCN encodes structural similarity, while fixed ResNet-152 features pass through a trainable layer to produce visual embeddings.The visual encoder is not fine-tuned, and the resulting representation is intended to capture low-level similarity and high-level semantic relatedness.
- Modality fusion: Trainable attention weights fuse row-normalised modality representations, with softmax-normalised weights summing to 1 across modalities.The model applies separate alignment losses to individual modalities and the merged representation, using the merged loss to train modality weights.
- Alignment learning: EVA computes cosine similarities for all cross-graph entity pairs and trains with NCA loss that uses local and global sample statistics while downweighting hard negatives.The joint loss combines modality-specific terms and a multimodal term, with distinct temperature settings for different feature distributions.
- Unsupervised visual pivoting: Visual pivot induction enables unsupervised alignment by selecting the most visually alike cross-graph entities without annotated pivots.Candidate pairs are proposed as mutual nearest neighbours and permanently added only after remaining mutual nearest neighbours for Ks consecutive rounds.
- Data setting and evaluation: KG-provided images are mostly human-verified and disambiguated, making visual pivoting less noisy than approaches based on web-crawled images.The table evaluates cross-lingual EA on DBP15k with and without iterative learning, including reproduced settings that remove extra machine translation or alignment labels.
4 Experiments
Experiments on DBP15k and DWY15k evaluate EVA across semi-supervised and unsupervised settings, ablations, visual encoders, and long-tail entities. EVA achieves strong alignment performance, while visual information particularly benefits entities with sparse structural context, though missing or inconsistent images limit gains.
- Main results: EVA surpasses baselines across semi-supervised settings, with 12.3-17.6% absolute H@1 gains without IL and 11.9-12.5% with IL on cross-lingual DBP15k.On monolingual DWY15k, EVA also exceeds the strongest baselines across normal and dense splits.
- Unsupervised visual seeding: H@1 exceeds 55% with 100 visual seeds and surpasses 70% with more than 3k seeds, but later declines as larger dictionaries introduce noise.A cosine similarity threshold of 0.85 is reported as a useful cutoff.
- Ablation study: Removing structural embeddings lowers H@1 from about 80% to below 40%, while removing images or attributes lowers H@1 by 4-5%.Relations cause an approximately 3% H@1 drop when removed; learned attention weights place the greatest weight on structure.
- Long-tail analysis: Visual representations benefit entity pairs in lower degree-centrality ranges more than higher-degree pairs, supporting their value for long-tail entities.The most frequent entities can perform worse than the second-most frequent split, indicating that denser neighborhoods do not always improve alignment.
- Limitations: Cross-lingual performance remains imperfect because 1,945 of 2,955 EVA errors involve entities without valid images, while inconsistent visual representations can also cause misalignment.For example, a university gate photo and a logo represent the same entity differently across languages.
5 Conclusion
EVA uses images as pivots and fuses multimodal KG information for entity alignment. Its unsupervised variant builds and expands seed dictionaries from visual similarities, achieving strong benchmark performance and highlighting benefits for long-tail entities.
- 5 Conclusion: EVA fuses images, structures, relations, and attributes through attention-based modality weighting into a joint entity representation.The model automatically adjusts modality weights during alignment.
- 5 Conclusion: EVA’s unsupervised variant builds a seed dictionary from visual similarities and expands it through iterative learning.This approach avoids requiring alignment labels.
- 5 Conclusion: >70% accuracy is achieved by unsupervised EVA, close to its semi-supervised performance and above the previous best semi-supervised baseline.
- 5 Conclusion: EVA’s ablation studies and error analysis indicate that images benefit alignment of long-tail KG entities.These entities inherently lack sufficient structural contexts for capturing correspondences.
A More Implementation Details
The implementation details specify hyper-parameter search and hardware settings used to balance EVA’s memory requirements and performance.
- A More Implementation Details: Hyper-parameter search balances memory limits against model performance rather than always selecting the best-performing model.The search space is listed in Table 8.
- A More Implementation Details: EVA’s full training process takes 10–15 minutes for 1,000 epochs on the reported machine.
- A More Implementation Details: The full model has approximately 16M trainable parameters on DBP15k and 13M on DWY15k.
B Full Table for the Unsupervised Setting Results
Table 9 presents the unsupervised-setting data from Figure 2 in tabular form.
- B Full Table for the Unsupervised Setting Results: Table 9 reproduces the data shown in Figure 2 as a table.
C Plottings of Normalised Modality Weights
Images receive substantial modality weight during training, ranking second on DBP15k and third on DWY15k for most of the process.
- C Plottings of Normalised Modality Weights: On DBP15k, images are the second-most important modality after graph structure throughout almost the entire training process.
- C Plottings of Normalised Modality Weights: On DWY15k (norm), images rank third after graph structure and surface form for almost the whole training process.
- C Plottings of Normalised Modality Weights: On DBP15k, image weight rises slightly after the starting phase and decreases once iterative learning begins.
D Descriptions of Baseline Methods
Cross-lingual entity-alignment baselines divide into methods with or without iterative learning, while monolingual work adds co-training over structural and attribute views.
- Cross-lingual baselines fall into two categories: methods with iterative learning and methods without it.
- MTRANSE jointly learns translational embeddings and an alignment model using transformations or distances between entity representations.
- GCN-, multi-channel GNN-, and relationally regularized methods improve entity representations by modeling neighborhood structure or relational translation.
- BOOTEA bootstraps MTRANSE, while MMEA and NAEA use GCN- and GAT-based approaches with NAEA adding a mutual-nearest-neighbor constraint for new labels.
- Monolingual alignment adds COTSAE, which iteratively co-trains structural and attribute views similarly to KDCOE.
E Future Research Directions
The paper identifies cross-language performance differences and extending EVA beyond high-resource languages as future research directions, while Figure 6 tracks normalized weights across epochs.
- Investigation of alignment difficulties across different language pairs: H@1 for FR→EN plateaus earlier than for JA→EN and ZH→EN, motivating analysis of language-pair differences.
- Investigation of alignment difficulties across different language pairs: Future work should examine how image distributions within each language influence cross-lingual mapping.
- Extending to low-resource languages: The study focuses on high-resource languages, leaving low-resource-language alignment as an important extension.
- Figure 6 plots normalized weights against the number of epochs.