Source-linked AI summary
Universal Representation Learning of Knowledge Bases by Jointly Embedding Instances and Ontological Concepts
Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, Wei Wang
TL;DR
Existing KG embedding models largely represent either instance or ontology knowledge, despite knowledge bases containing both views and often only sparse links between them. JOIE jointly models intra-view structure and cross-view associations, and experiments report stronger performance across KG completion, ontology population, and entity typing. The paper concludes that JOIE captures useful latent features from both views and outperforms state-of-the-art baselines.
Problem
Existing KG embedding models focus on one view, while jointly representing instance entities, ontology concepts, and their sparsely covered cross-view links remains challenging.
Method
JOIE jointly embeds instance and ontology views with separate intra-view models, cross-view grouping or transformation, and optional hierarchy-aware encoding for ontology hierarchies.
Results
JOIE outperforms baselines on KG completion and entity typing, including 15.4% accuracy and 14.4% MRR gains over MTransE on YAGO26K-906.
Takeaways & Limitations
Jointly capturing structured knowledge within both views and their cross-view links supports knowledge embedding, ontology population, and entity typing across the evaluated datasets.
Takeaways & Limitations
JOIE cannot directly predict previously unseen meta-relations that are absent from the meta-relation vocabulary.
Abstract
from arXiv · showhide
Many large-scale knowledge bases simultaneously represent two views of knowledge graphs (KGs): an ontology view for abstract and commonsense concepts, and an instance view for specific entities that are instantiated from ontological concepts. Existing KG embedding models, however, merely focus on representing one of the two views alone. In this paper, we propose a novel two-view KG embedding model, JOIE, with the goal to produce better knowledge embedding and enable new applications that rely on multi-view knowledge. JOIE employs both cross-view and intra-view modeling that learn on multiple facets of the knowledge base. The cross-view association model is learned to bridge the embeddings of ontological concepts and their corresponding instance-view entities. The intra-view models are trained to capture the structured knowledge of instance and ontology views in separate embedding spaces, with a hierarchy-aware encoding technique enabled for ontologies with hierarchies. We explore multiple representation techniques for the two model components and investigate with nine variants of JOIE. Our model is trained on large-scale knowledge bases that consist of massive instances and their corresponding ontological concepts connected via a (small) set of cross-view links. Experimental results on public datasets show that the best variant of JOIE significantly outperforms previous models on instance-view triple prediction task as well as ontology population on ontologyview KG. In addition, our model successfully extends the use of KG embeddings to entity typing with promising performance.
1 INTRODUCTION
Large knowledge bases contain instance and ontology views linked by cross-view associations, but existing embedding models generally represent only one view. JOIE jointly models both views to capture their complementary information and address their differing vocabularies, coverage, and structure.
- Two-view knowledge bases: Knowledge bases organize knowledge into instance-view graphs of specific entities and ontology-view graphs of abstract concepts, connected by links indicating instance–concept membership.The ontology view also contains semantic meta-relations and hierarchical meta-relations.
- Motivation: KG embeddings encode graph structure in low-dimensional spaces and support relational inference and knowledge-driven tasks, but existing models focus on only one view.The passage lists applications including relation extraction, question answering, dialogue agents, knowledge alignment, and visual semantic labeling.
- Motivation: Joint representation can enrich concept embeddings with detailed instance information while providing sparse instances with high-level information from their concepts.This is especially useful when an instance has few relational facts in the instance-view graph.
- Challenges: Combining the views is difficult because their vocabularies are disjoint but related, cross-view links are sparse, and their scales and topologies differ.Ontology graphs are typically smaller, sparser, and more hierarchical, whereas instance graphs are larger and contain more relation types.
- JOIE: JOIE jointly embeds both views using triples from each graph and cross-view links connecting entities to concepts.Its design targets joint representation despite the semantic and structural differences between the two views.
2 RELATED WORK
Related work covers instance-view KG embeddings, methods incorporating entity types, and ontology population, while distinguishing JOIE’s joint treatment of ontology semantics and cross-view information.
- Knowledge Graph Embeddings: Knowledge graph embedding research learns plausibility functions for triples in instance-view graphs to represent relations among entities.The cited related-work passage introduces triples (h,r,t) and scoring functions as the basis of KG embeddings.
- Knowledge Graph Embeddings: Methods that incorporate entity types strengthen instance-level entity similarity but generally do not model semantic relations among the types.These approaches also focus on instance-view triple completion rather than using instance knowledge for ontology population or cross-view association.
- Ontology Population: Traditional ontology population relies on manual effort or large annotated text corpora, with parsing and human-generated facts creating scalability and conflict concerns.The related-work passage characterizes these requirements as intractable or prone to frequent conflicts.
3 MODELING
JOIE jointly models instance and ontology views through cross-view association and separate intra-view embedding spaces. Its variants support grouping or transformation across views, triple-based structural encoding, and hierarchy-aware ontology modeling.
- Model overview: JOIE jointly embeds entities and concepts using cross-view association and intra-view models.The cross-view component connects corresponding instances and concepts, while intra-view models encode each graph's structure.
- Cross-view association: Cross-view Grouping places corresponding entity and concept embeddings in one space and penalizes distances beyond margin γCG.It requires equal embedding dimensionalities and creates a clustering effect around corresponding concepts.
- Cross-view association: Cross-view Transformation aligns separate entity and concept spaces by mapping entity embeddings toward corresponding concept embeddings.Unlike grouping, transformation permits completely different embedding spaces and uses a nonlinear affine mapping.
- Intra-view modeling: The default intra-view model learns structural representations from triples separately in the instance and ontology graphs.It applies score functions and hinge loss with corrupted triples, using translation, multiplication, or circular-correlation techniques.
- Intra-view modeling: The hierarchy-aware intra-view model adds nonlinear transformations and hierarchy loss for ontology concept pairs.This extends structural encoding to hierarchical meta-relations while retaining regular semantic-relation modeling.
- Variants: JOIE provides six non-hierarchy variants and three additional hierarchy-aware variants combining two association methods with three intra-view encoders.Examples include JOIE-TransE-CT and hierarchy-aware variants based on TransE, multiplication, and HolE.
4 EXPERIMENTS
Experiments evaluate JOIE on two new two-view datasets across triple completion, entity typing, ontology population, and long-tail prediction. JOIE generally outperforms baselines, with benefits varying by cross-view, intra-view, and ontology-structure settings.
- Experimental setup: The experiments use YAGO26K-906 and DB111K-174 for instance-view and ontology-view triple completion, entity typing, ontology population, and long-tail entity typing.The datasets differ in entity-to-concept link density, with DB111K-174 having a much higher ratio than YAGO26K-906.
- KG triple completion: JOIE outperforms comparable baselines by 6.8% on MRR and 14.8% on Hit@10 on average for triple completion.On DB111K-174 ontology completion, JOIE improves 10.4% over “all”-setting baselines and 34.97% over “base”-setting baselines.
- Entity typing: JOIE-TransE-CT improves entity-typing accuracy and MRR over MTransE by 15.4% and 14.4% on YAGO26K-906, and 14.3% and 14.5% on DB111K-174.The results indicate that cross-view links require a dedicated representation technique rather than treatment as a regular relation.
- Model variants: JOIE-TransE-CT is consistently the strongest variant, gaining 4.1% MRR over JOIE-HolE-CT and JOIE-DistMult-CT and 2.17% accuracy over JOIE-TransE-CG.CT can use different embedding dimensions and is less sensitive to structural inconsistency between the two views.
- Hierarchy-aware modeling: Hierarchy-aware CT improves entity typing by 6.0% on MRR and 10.5% on accuracy on average, especially for YAGO26K-906’s relatively rich ontology.The same effect is not observed on DB111K-174 because its ontology has much smaller hierarchical structures.
- Case studies: JOIE remains more robust on long-tail entities, whose MRR declines by 12.5% with CG and 12.3% with CT, compared with over 20% for other baselines.Smaller embeddings benefit long-tail entity typing, while the authors hypothesize that larger dimensions can overfit entities with limited training data.
5 CONCLUSION AND FUTURE WORK
JOIE jointly embeds real-world entities and ontological concepts, capturing structured knowledge within both views and links between them. The paper reports strong performance on KG completion and entity typing, while identifying higher-order structure and relation alignment as future directions.
- JOIE jointly embeds real-world entities and ontological concepts while capturing structured knowledge within each view and links between views.
- Extensive experiments on KG completion and entity typing show that JOIE captures latent features from both views and outperforms state-of-the-art baselines.
- Future work will use higher-order neighborhoods, logic paths, or global graph structures instead of only local triple-based structure loss.
- Future work will also explore alignment between relations and meta-relations, analogous to entity-concept alignment.
A ABLATION STUDY
The ablation study examines embedding dimensionality, the sufficiency of cross-view links, and negative sampling in cross-view association models as factors affecting JOIE's performance.
- The ablation study evaluates embedding dimensionality, cross-view-link sufficiency, and negative sampling in cross-view association models.
A.1 Dimensionality
Embedding dimensionality affects JOIE's performance, with the best setting depending on the cross-view association technique and dataset. Moderate dimensions can outperform both lower dimensions and overly high dimensions.
- CG variants generally improve from d = 50 to d = 200, then their MRR drops at d = 300.
- Most JOIE variants perform best with embedding dimensions (d1, d2) = (300, 50), rather than (300, 100).
- Low dimensionalities may fail to capture latent entity and concept features, whereas excessively high dimensionalities can overfit the ontology-view KG.
A.2 Sufficiency of Type Information
Cross-view links bridge the instance and ontology views and support information flow between them. Increasing their training proportion improves MRR overall, while CG and CT differ in sensitivity to additional links.
- Cross-view links bridge instance-view and ontology-view graphs and enable information flow for generating embeddings.
- Increasing training links from 20% to 80% improves MRR by 3.2% on YAGO26K-906 and 2.9% on DB111K-174.
- When the training proportion rises from 0.6 to 0.8, CG performance does not necessarily improve, whereas CT performance still improves significantly.
- The authors hypothesize that CG is sensitive to structural inconsistencies between views, while CT is more robust, creating a robustness-efficiency trade-off.
A.3 Effects of Negative Sampling
The study evaluates whether negative sampling should be applied to cross-view entity–concept links. It finds that the effect depends on the JOIE variant and improves neighbor quality in a representative query.
- A.3 Effects of Negative Sampling: Negative sampling is tested for cross-view links by corrupting entity–concept pairs such as (“Barack Obama”,“state”).The comparison evaluates JOIE variants with and without cross-view link negatives on entity typing.
- A.3 Effects of Negative Sampling: Table 7 reports the effects of negative sampling on type links.The table is the quantitative comparison referenced for entity typing.
- A.3 Effects of Negative Sampling: Disabling negative sampling causes a significant performance drop in JOIE-TransE-CT, whereas it has less effect on JOIE-TransE-CG.The authors attribute CG’s smaller effect to its strong clustering-based constraint, which already separates irrelevant concepts.
- A.3 Effects of Negative Sampling: Without negative sampling, JOIE-TransE-CT retrieves irrelevant concepts such as “decoration” and “architect” among the nearest neighbors of “music”.Related concepts including “classic music”, “concert” and “artist movement” remain close to “music” in both settings.
B DATASETS
The datasets are built by filtering relational triples, aligning entities with ontology concepts, and extracting corresponding ontology subgraphs. YAGO’s ontology is enriched with ConceptNet knowledge, producing datasets larger than FB15K while retaining long-tail frequency patterns.
- B DATASETS: YAGO26K-906 and DB111K-174 are extracted from connected subsets of YAGO and DBpedia, respectively.These datasets are used for the experiments.
- B DATASETS: Attribute triples are removed, and relational triples are sampled from the remaining data to form the instance-view entity sets.The filtering excludes triples that do not represent relations among entities or concepts.
- B DATASETS: Entities are aligned to associated ontology concepts, and the ontology view is constructed from the intersecting subgraph of the original ontologies.The associated concepts become nodes in the ontology view.
- B DATASETS: YAGO’s ontology is enriched with ConceptNet because its original taxonomy has only three semantic relation types.ConceptNet and YAGO concepts are aligned through shared WordNet-based IDs or concept names.
- B DATASETS: The resulting datasets are much larger than FB15K, while entity and relation frequencies follow long-tail distributions in both datasets.The long-tail pattern is confirmed by the histogram in Figure 8.