Source-linked AI summary
Embedding Multimodal Relational Data for Knowledge Base Completion
Pouya Pezeshkpour, Liyan Chen, Sameer Singh
TL;DR
Existing knowledge-base embeddings mainly model links among fixed entities, leaving text, images, and numerical values underused. MKBE adds modality-specific encoders and decoders to support multimodal link prediction and missing-value generation, achieving state-of-the-art link-prediction results and producing generated attributes evaluated as realistic and informative. The paper also identifies limited dataset scale and sparse embeddings as constraints on generated-attribute quality.
Problem
Existing relational embeddings focus on links among fixed entities, while knowledge bases also contain textual, image, numerical, and categorical information that can support completion.
Method
MKBE combines modality-specific neural encoders with relational scoring models and uses neural decoders conditioned on entity embeddings to generate missing multimodal attributes.
Results
MKBE achieves higher link-prediction accuracy than DistMult and ConvE by using multimodal evidence and generates high-quality images and text evaluated through user studies.
Takeaways & Limitations
The model provides a unified way to use heterogeneous knowledge-base information for link prediction and multimodal attribute imputation.
Takeaways & Limitations
Generated attributes remain far from ideal because the datasets are much smaller than common generation datasets and the knowledge graphs are sparse.
Abstract
from arXiv · showhide
Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, images, and numerical values. In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data. Further, using these learned embedings and different neural decoders, we introduce a novel multimodal imputation model to generate missing multimodal values, like text and images, from information in the knowledge base. We enrich existing relational datasets to create two novel benchmarks that contain additional information such as textual descriptions and images of the original entities. We demonstrate that our models utilize this additional information effectively to provide more accurate link prediction, achieving state-of-the-art results with a considerable gap of 5-7% over existing methods. Further, we evaluate the quality of our generated multimodal values via a user study. We have release the datasets and the open-source implementation of our models at https://github.com/pouyapez/mkbe
1 Introduction
Knowledge bases contain multimodal information beyond links, motivating MKBE, which encodes these data types for link prediction and multimodal value generation. The paper evaluates this approach on two enriched benchmarks and reports improved link prediction and generated-attribute quality.
- Motivation: Knowledge bases include links alongside numerical, textual, and image attributes that can provide evidence for completing missing facts.Examples include using images to predict occupation and descriptions to infer nationality.
- Approach: MKBE extends relational embedding models with neural encoders for multimodal evidence and decoders that generate missing attributes such as text and images.The scoring module remains that of the underlying relational model, including DistMult or ConvE.
- Evaluation: The paper introduces two benchmarks by extending YAGO-10 and MovieLens-100k with textual descriptions, numerical attributes, and images.These benchmarks support evaluation in the multimodal knowledge-base completion setting.
- Results: MKBE achieves state-of-the-art link-prediction results on the enriched datasets and generates multimodal attributes judged realistic and informative in user studies.The evaluation includes both link-prediction accuracy and the quality of generated text and images.
2 Multimodal KB Completion
The multimodal completion framework treats text, images, numerical values, and categorical values as objects that can be encoded, scored, and recovered. MKBE combines domain-specific encoders for relational scoring with decoders that generate missing multimodal values.
- Problem formulation: Multimodal knowledge-base completion scores triples whose objects may be entities or values such as images, text, numbers, and categories.The model also predicts missing objects, including multimodal values, rather than only missing links between entities.
- Problem formulation: Existing entity-indexed embeddings cannot represent multimodal content or generalize their learned object vectors to unseen attribute values.This motivates content-based encoders for multimodal objects.
- Encoding: MKBE uses domain-specific encoders to map structured, numerical, textual, and image objects into the embedding space used by relational scoring models.Numerical values are projected from R to R^d, while text uses character-based bidirectional GRUs for short strings and images use modality-specific representations.
- Scoring and training: Entity and relation embeddings are combined with encoded object representations to score the truth of each multimodal triple.The approach retains the scoring functions of models such as DistMult and ConvE.
- Imputation: Neural decoders use learned entity embeddings as context to generate missing multimodal values after encoding alone has provided representations.The decoder mapping generates values in the multimodal space M for missing-value queries.
3 Related Work
Prior relational embedding methods primarily model fixed entity links, whereas MKBE treats multimodal information as relational data within a unified completion model. The paper distinguishes this design from approaches that use additional information only as auxiliary entity features.
- Existing relational embeddings: Existing relational embedding approaches differ mainly in their scoring operators but generally represent only structured links among a fixed set of entities.Their objects are treated as members of an enumerable entity set.
- Benchmarks: The evaluation uses extended benchmark datasets whose added multimodal statistics are presented as contributions to the datasets.Table 1 reports data statistics for the two benchmark datasets.
- MKBE distinction: MKBE treats text, numerical values, and images as relational triples rather than merely as features attached to entities.This makes multimodal information part of the modeled knowledge base.
- MKBE distinction: The paper identifies three distinctions from related approaches: unified multimodal modeling, first-class treatment of heterogeneous data, and support for uncertainty and missing-value recovery.These distinctions cover numerical, textual, and image information within one model.
4 Evaluation Benchmarks
The paper extends MovieLens-100k and YAGO3-10 with multimodal and attribute information to benchmark multimodal knowledge-base completion.
- Two benchmarks extend existing relational datasets with additional multimodal information for evaluating knowledge-base completion.The paper provides dataset statistics for these extensions.
- MovieLens-100k contains about 1,000 users and 1,700 movies, with ratings, user attributes, movie genres, release dates, titles, and collected posters.The five-point ratings are represented as five separate knowledge-base relations.
- YAGO3-10 provides a larger knowledge-graph benchmark with about 120,000 entities and 37 relations spanning people, locations, and organizations.It is intended to be more suitable for general knowledge-graph completion than the smaller, specialized MovieLens dataset.
- YAGO-10 link-prediction results compare structured information, textual descriptions, dates, images, and published results from Dettmers et al. (2018).The table distinguishes models by the information types they use.
5 Experiment Results
Experiments evaluate multimodal information for link prediction and missing-value generation across text, images, numerical attributes, categories, and movie titles. Models using broader information generally perform better, while the most useful modality varies by relation and generated-value task.
- Evaluation setup: MKBE is evaluated against DistMult and ConvE on link prediction and against decoders for missing text, images, numerical values, categories, and titles.The link-prediction evaluation uses ranking metrics, while generation quality is assessed through user studies and numerical accuracy.
- Link prediction: On MovieLens, R+M+U+T outperforms other models by a considerable gap, and titles provide more benefit than posters.The baseline Hits@1 is 40%.
- Link prediction: On YAGO, models using all information consistently outperform other variants, while text-only models rank second and structured-only models are outperformed by every alternative.The pattern holds for both DistMult and ConvE, with ConvE results reported as state of the art for the dataset.
- Link prediction: Textual descriptions benefit isAffiliatedTo and playsFor, images help hasGender and isMarriedTo, and dates are more effective than images for isConnectedTo.The relation-level comparison shows that modality usefulness depends on the relation.
- Attribute imputation: Using all information improves prediction of numerical dates and MovieLens genres, while neural decoding outperforms search-based decoding for numerical values.Dates are evaluated on held-out values more recent than 1000AD, and genres use 10% held-out MovieLens data.
- Text generation: Titles generated from embeddings using all information are judged more similar to reference movie titles than titles conditioned only on ratings.Users assessed title realism and genre appropriateness in a study using three participants per question.
- Text generation: Descriptions generated from embeddings with more information outperform descriptions generated from structured data alone in conveying entity information.The evaluation uses short YAGO descriptions and user judgments.
- Image generation: Images generated from embeddings using all information are more accurate for gender and occupation, but age is difficult to infer because source photographs may not match a person’s age.The image evaluation uses user judgments over generated examples.
6 Discussion and Limitations
MKBE extends relational embedding approaches to multimodal knowledge bases, but its scalability and generated-attribute quality remain constrained by training cost, overfitting, dataset size, and sparse embeddings.
- Scalability: Multimodal encoders and decoders add training expense, but the scalability cost remains additive because multimodal information is treated as additional triples.The approach is described as not significantly worse than existing embedding-based link prediction techniques.
- Multimodal modeling: Selecting informative multimodal evidence is difficult, and the models remain prone to overfitting despite demonstrating effective information use.The paper calls for more efficient and accurate models for capturing multimodal attributes.
- Generated attributes: Generated attributes are limited by GAN capability and the information captured in embedding vectors.The imputation model uses adversarial generation from embeddings learned for knowledge-base representation.
- Generated attributes: The generated text and images are far from ideal because both datasets are much smaller than common generation datasets and the knowledge graphs are sparse.The paper proposes expanding multimodal datasets and using the knowledge graph directly for generation in future work.
7 Conclusion
The paper presents MKBE for jointly learning relational and multimodal representations, using them both for link prediction and for imputing missing attributes. On enriched YAGO-10 and MovieLens-100k benchmarks, MKBE improves link-prediction accuracy over DistMult and ConvE and generates multimodal attributes such as images and text.
- Approach: MKBE jointly learns entity and multimodal embeddings, then uses adversarial decoding to impute missing multimodal values.The encoding component combines information available for each entity, while the decoding component generates missing attributes from entity embeddings.
- Benchmarks: MKBE enriches YAGO-10 and MovieLens-100k with multimodal information to create benchmarks for evaluating link prediction and attribute imputation.The added information supports experiments involving links, images, text, and other multimodal attributes.
- Findings: MKBE achieves higher link-prediction accuracy than DistMult and ConvE by utilizing multimodal evidence.The paper also reports that the model generates high-quality multimodal attributes such as images and text.