Source-linked AI summary
Image-embodied Knowledge Representation Learning
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, Maosong Sun
TL;DR
Conventional knowledge representation learning mainly uses structured triples, leaving visual information in entity images underused. IKRL encodes and attention-aggregates entity images alongside triple facts, and its models achieve state-of-the-art results on knowledge graph completion and triple classification. The paper concludes that image information can be encoded into knowledge representations, while noting limitations in image features, the TransE basis, and single-entity image modeling.
Problem
Conventional knowledge representation learning methods focus on structured triple facts and ignore rich visual information in entity images.
Method
IKRL uses a neural image encoder, attention-based aggregation of an entity’s image instances, and joint translation-based learning with structured knowledge.
Results
IKRL achieves state-of-the-art performance on both knowledge graph completion and triple classification, outperforming baselines across reported combination strategies.
Takeaways & Limitations
The results support the significance of visual information and the capability of IKRL to encode image information into knowledge representations.
Takeaways & Limitations
The current models depend on image representation quality and TransE, and treat each image as representing only its target entity rather than multiple entities and relations.
Abstract
from arXiv · showhide
Entity images could provide significant visual information for knowledge representation learning. Most conventional methods learn knowledge representations merely from structured triples, ignoring rich visual information extracted from entity images. In this paper, we propose a novel Image-embodied Knowledge Representation Learning model (IKRL), where knowledge representations are learned with both triple facts and images. More specifically, we first construct representations for all images of an entity with a neural image encoder. These image representations are then integrated into an aggregated image-based representation via an attention-based method. We evaluate our IKRL models on knowledge graph completion and triple classification. Experimental results demonstrate that our models outperform all baselines on both tasks, which indicates the significance of visual information for knowledge representations and the capability of our models in learning knowledge representations with images.
1 Introduction
Knowledge representation methods largely rely on structured triples despite entity images containing visual information. IKRL incorporates image representations with triple-based learning and achieves state-of-the-art performance on knowledge graph completion and triple classification.
- Knowledge graphs represent entities and relations through structured triple facts, supporting knowledge inference and question answering.
- Conventional knowledge representation learning methods focus on triple facts while disregarding rich information in entity images.
- IKRL encodes each entity image with a neural encoder, aggregates multiple image representations through attention, and jointly learns representations with translation-based methods.
- IKRL is evaluated on knowledge graph completion and triple classification, achieving state-of-the-art performance on both tasks.
- The work combines visual information with knowledge graphs and analyzes attention-based image selection and semantic regularities between image representations.
2 Related Work
Translation-based methods learn entity and relation embeddings by treating relations as translations, while later variants address limitations of the basic assumption. Multi-source approaches incorporate textual or visual information, but IKRL explicitly brings images into knowledge representation learning.
- TransE embeds entities and relations in a shared low-dimensional space, assuming the tail entity is near the head entity plus the relation.
- TransH, TransR, and TransD extend translation-based learning to model complicated entities and relations through hyperplanes, separate spaces, or dynamic mappings.
- Prior multi-source methods use textual or visual information for tasks including image-sentence ranking, metaphor identification, and visual question answering.
- IKRL addresses the stated gap by explicitly encoding visual information from images into knowledge representations.
3 Methodology
IKRL combines structure-based and image-based entity representations in a translation-based energy model. It encodes each image, projects features into entity space, and uses attention to aggregate multiple images before margin-based training.
- 3.1 Overall Architecture: IKRL represents each entity with both structure-based embeddings learned from triples and image-based representations constructed from corresponding images.
- 3.1 Overall Architecture: The overall energy combines structure–structure, structure–image, image–structure, and image–image interactions.The components are ESS, ESI, EIS, and EII.
- 3.2 Image Encoder: The image encoder extracts neural image features and projects them from image space into entity space using a shared projection matrix.The projection maps di-dimensional image features to ds-dimensional entity representations.
- 3.3 Attention-based Multi-instance Learning: Attention aggregates multiple image representations by assigning greater weight to images more similar to the entity’s structure-based representation.AVG instead weights all images equally, while MAX retains only the representations with the highest attention.
- 3.4 Objective Formalization: The model uses a margin-based objective with corrupted triples as negatives and optimizes parameters using mini-batch stochastic gradient descent.Negative triples are generated by randomly replacing an entity or relation and removing replacements already present in the training set.
4.1 Dataset
The authors construct WN9-IMG, a knowledge graph dataset paired with entity images for knowledge graph completion and triple classification. It derives triples from WN18 and associates every entity with ImageNet images.
- 4.1 Dataset: WN9-IMG is a newly constructed knowledge graph–image dataset evaluated on knowledge graph completion and triple classification.
- 4.1 Dataset: The triple portion of WN9-IMG is a subset of WN18, which was originally extracted from WordNet.
- 4.1 Dataset: The dataset uses 63,225 ImageNet images and ensures that every entity has images.Extracted triples are randomly split into training, validation, and test sets.
4.2 Experiment Settings
The experiments train IKRL with mini-batch SGD and compare it with TransE and TransR under matched embedding dimensions. The selected configuration uses margin 4.0 and a linearly declining learning rate.
- 4.2 Experiment Settings: IKRL is trained with mini-batch SGD, testing margin values {1.0, 2.0, 4.0} and learning rates {0.0002, 0.0005, 0.001} or an adaptive schedule.
- 4.2 Experiment Settings: The optimal configuration uses γ = 4.0 with a linearly declining learning rate from 0.001 to 0.0002.
- 4.2 Experiment Settings: TransE and TransR serve as baselines, with entity and relation dimensions set to 50 for all baselines.
4.3 Knowledge Graph Completion
IKRL models are evaluated for entity prediction using Mean Rank and Hits@10, with variants combining structure-based and image-based representations through different image-combination strategies. Across the reported comparisons, IKRL outperforms baselines, and attention-based aggregation performs best.
- Evaluation protocol: Entity prediction evaluates missing-entity completion using the dissimilarity function ||h + r − t||, reporting Mean Rank and Hits@10 under Raw and Filter settings.IKRL (SBR) uses structure-based representations, while IKRL (IBR) uses image-based representations for prediction.
- Entity prediction results: All IKRL models outperform all baselines on Mean Rank and Hits@10, with IKRL (UNION) achieving the best performance.The results are reported in Table 3.
- Entity prediction results: IKRL (SBR) and IKRL (IBR) both outperform baselines, indicating that visual information improves image-based and structure-based entity representations.The paper attributes the stronger Mean Rank results to image information supplementing missing information in structured triples.
- Attention analysis: IKRL (ATT) achieves the best performance among the combination strategies, while IKRL (AVG) outperforms IKRL (MAX).The comparison suggests that jointly weighting multiple images preserves more information than selecting only the highest-attention image.
- Attention analysis: IKRL models outperform baselines across combination strategies on both Mean Rank and Hits@10, confirming that image information is encoded into knowledge representations.The attention model has only slight advantages over averaging when extracted ImageNet images are already high quality.
4.4 Triple Classification
Triple classification predicts whether a knowledge-graph triple is correct using relation-specific thresholds over a dissimilarity function. IKRL variants outperform both baselines, with attention-based aggregation performing best.
- Evaluation protocol: Triple classification is a binary task that predicts whether a triple fact is correct, with negative instances generated by randomly replacing head or tail entities.The number of positive and negative triples is balanced.
- Evaluation protocol: Relation-specific thresholds δ_r classify triples as negative when ||h + r − t|| exceeds δ_r and positive otherwise.The thresholds are optimized on the validation set for their corresponding relations.
- Evaluation protocol: Only image-based representations are used when calculating the triple-classification dissimilarity function.This setting is used to better demonstrate the advantages of IKRL models.
- Results: All IKRL models outperform both baselines, including the enhanced TransR model, demonstrating effectiveness when structured triples and visual information are combined.The comparison is reported in Table 4.
- Results: IKRL (ATT) performs best among the combination strategies by jointly considering multiple images and selecting more informative instances.The result supports attention-based aggregation for triple classification.
4.5 Case study
The case study examines semantic regularities in image-based representations and attention-based selection among multiple entity images. It reports interpretable translation patterns and examples where attention downweights noisy or misleading images.
- Case study: The case study investigates semantic regularities of images and the capability of attention to select informative instances.Images may be cropped for presentation, while their main objects remain included.
- Semantic regularities of images: In the image-knowledge joint space, dresser minus drawer matches the concrete relation part of, making the translation regularity interpretable.This extends the discussion of semantic regularities from word and image-text representations to image-based knowledge representations.
- Attention examples: Attention assigns low weight to a phone mistakenly included among portable-computer images, identifying it as a low-quality instance.The example illustrates attention-based discrimination among multiple candidate images.
- Attention examples: For golf game, an overview of the lawn receives low attention because it lacks people or detailed sporting goods.The image is less useful for representing the whole entity.
- Attention examples: For watering pot, attention downweights an image focused on the spout because it may confuse representation of the whole entity.The authors state that attention helps learn representations from better images while alleviating noise from multiple instances.
5 Conclusion and Future Work
IKRL learns knowledge representations from entity images using neural image modeling, projection, and attention-based aggregation. The authors report successful image encoding while identifying image quality, model extensions, and multi-entity images as future work.
- Conclusion: IKRL uses neural networks and a projection module to model each image, then combines multiple image instances through attention.This produces aggregated image-based representations for entities.
- Conclusion: Experimental results confirm that IKRL models can encode image information into knowledge representations.
- Future Work: Image-representation quality is identified as essential, motivating more sophisticated feature-extraction models for specific domains.
- Future Work: The authors plan to extend IKRL beyond TransE to other enhanced translation-based methods.
- Future Work: Current models treat each image as representing only its target entity, although images may contain multiple entities and their relations.Future work proposes learning those entities and relations jointly with IKRL.