Source-linked AI summary
TransG : A Generative Mixture Model for Knowledge Graph Embedding
Han Xiao, Minlie Huang, Yu Hao, Xiaoyan Zhu
TL;DR
Knowledge graph embedding lacks a formal treatment of relations with multiple meanings, even though entity-pair patterns reveal such semantics. TransG introduces a generative Bayesian non-parametric infinite mixture model that discovers latent relation semantics and combines relation components, with experiments showing substantial improvements over state-of-the-art baselines. Its evaluation also identifies poor Mean Rank cases associated with insufficient training data.
Problem
Knowledge graph embedding must address relations whose associated entity pairs reveal multiple meanings, a limitation not formally studied by previous models.
Method
TransG uses a generative Bayesian non-parametric infinite mixture model to discover latent relation semantics and leverage mixtures of relation components for embedding.
Results
Extensive experiments show substantial improvements against state-of-the-art baselines, including 2.9% improvement over TransR on WN18 and 26.0% on FB15K.
Takeaways & Limitations
TransG's results support capturing multiple relation semantics as beneficial for knowledge graph embedding.
Takeaways & Limitations
A small set of testing triples has poor Mean Rank because the corresponding relations and entities lack sufficient training data.
Abstract
from arXiv · showhide
Recently, knowledge graph embedding, which projects symbolic entities and relations into continuous vector space, has become a new, hot topic in artificial intelligence. This paper addresses a new issue of multiple relation semantics that a relation may have multiple meanings revealed by the entity pairs associated with the corresponding triples, and proposes a novel Gaussian mixture model for embedding, TransG. The new model can discover latent semantics for a relation and leverage a mixture of relation component vectors for embedding a fact triple. To the best of our knowledge, this is the first generative model for knowledge graph embedding, which is able to deal with multiple relation semantics. Extensive experiments show that the proposed model achieves substantial improvements against the state-of-the-art baselines.
1 Introduction
Knowledge graph embedding converts symbolic knowledge bases into continuous vectors, but prior translation-based models assign one vector per relation despite evidence that relations can have multiple latent meanings. TransG addresses this issue with an infinite mixture model that discovers semantic clusters, combines relation components, and improves over state-of-the-art baselines.
- Knowledge bases such as Wordnet and Freebase support AI tasks, but their symbolic and logical form limits direct use of numerical machine learning methods.
- Knowledge graph embedding projects entities and relations into continuous vector spaces, including translation-based models such as TransE, TransH, and TransR.
- Visualization of TransE embeddings reveals multiple clusters for individual relations, indicating different latent semantics rather than one relation-centered cluster.
- Previous translation-based models use one translation vector per relation, making them unable to represent multiple relation semantics and causing more errors for relations such as HasPart.
- TransG is a Bayesian non-parametric infinite mixture embedding model that automatically discovers semantic clusters and uses multiple relation components to translate entity pairs.
- Extensive experiments report substantial improvements over state-of-the-art baselines.
2 Related Work
Related work includes translation-based and other knowledge graph embedding methods. Translation-based approaches share a common translation principle, while later models add relation-specific projections, clustering, weighting, path information, or probabilistic representations.
- Prior embedding approaches are divided into translation-based methods and other approaches.
- Translation-Based Embedding Methods: Translation-based methods model triples with the principle h + r ≈ t, using relation-specific projections for entities in some models.
- Translation-Based Embedding Methods: TransR transforms entity embeddings with a relation-specific matrix, while CTransR clusters entity pairs and assigns one relation vector to each cluster.
- Pioneering Embedding Methods: Other approaches include Structured Embedding, SME, SLM, LFM, NTN, UM, RESCAL, SSE, text-integrated models, rule-aware models, and path-based PTransE.
- Pioneering Embedding Methods: These methods use varied mechanisms, including entity-specific matrices, tensor or neural interactions, quadratic forms, matrix factorization, semantic smoothing, text, rules, and paths.
3 Methods
TransG models a relation with multiple semantic components rather than one fixed translation, using a Bayesian non-parametric mixture whose components are learned from data. Its geometric procedure selects the primary component for each triple, while training jointly optimizes embeddings, mixing factors, and variances.
- TransG: A Generative Model for Embedding: TransG uses a Bayesian non-parametric infinite mixture model to represent multiple semantic components for each relation.The Chinese Restaurant Process generates components when necessary and learns their number adaptively from data.
- TransG: A Generative Model for Embedding: The model draws entity embedding means from a standard normal prior and relation components from a Chinese Restaurant Process.The supplied generative process specifies a standard normal prior for each entity and CRP sampling for relation components.
- TransG: A Generative Model for Embedding: Each relation component represents a latent meaning, with mixing factors indicating component weights and the number of components learned automatically.The mixing factor πr,m weights component m, while Mr is learned from the data through the CRP.
- Explanation from the Geometry Perspective: For a given triple, TransG identifies a primary component and translates the head entity to the tail using that component’s vector.Although all components contribute, the primary component contributes most because of the exponential effect.
- Training Algorithm: The training objective maximizes the likelihood ratio of true to false triples while jointly learning mixing factors and variances.SGD solves the optimization, and updates may be skipped for very impossible triples under a stated condition.
- Training Algorithm: TransG has time complexity O(M × O(TransE)) and showed no significant convergence-speed difference from TransE in the link-prediction experiment.M is the number of semantic components, and both methods converged at around 500 epochs.
4 Experiments
Experiments evaluate TransG on link prediction, triple classification, and semantic-component analysis across WordNet and Freebase benchmarks. TransG outperforms baselines while modeling multiple semantics, though sparse entity–relation evidence harms a small number of link-prediction cases.
- Experimental setup: Experiments use four WordNet and Freebase benchmark datasets for link prediction, triple classification, and semantic-component analysis.Link prediction uses WN18 and FB15K; triple classification uses WN11 and FB13.
- 4.1 Link Prediction: Link prediction ranks corrupted entities using Mean Rank and HITS@10 under Raw and Filter settings.Lower Mean Rank and higher HITS@10 indicate better performance; Filter removes known triples from ranking.
- 4.1 Link Prediction: 2.9% on WN18 and 26.0% on FB15K: TransG improves over TransR in link prediction, with average component counts of 5.67 and 8.77.The reported improvements are accompanied by automatically learned multiple semantic components.
- 4.1 Link Prediction: 24 testing triples, or 0.5% of the set, have ranks above 30,000 and cause about 150 Mean Rank loss.Twenty-three involve entities that never co-occurred with the corresponding relations during training, indicating insufficient training data for those cases.
- 4.1 Link Prediction: TransG’s FB15 HITS@10 rises from 81.8% to 88.2% across dimensions 100 to 400, while TransE ranges from 47.1% to 51.3%.The reported values are 81.8%, 84.0%, 85.8%, and 88.2% for TransG, versus 47.1%, 48.5%, 51.3%, and 49.2% for TransE.
- 4.2 Triple Classification: 1.7% on WN11 and 5.8% on FB13: TransG improves over TransR in triple classification, with average component counts of 2.63 and 4.53.Triple classification predicts whether a triple is correct using relation-specific thresholds selected on validation data.
- 4.2 Triple Classification: Relations with more semantic components, such as “Synset Domain” and “Type Of”, improve more than “Similar”, which has one component.This pattern further supports the reported benefit of capturing multiple relation semantics.
- Semantic component analysis: Most relations require multiple semantic components; “Also See”, “Synset Usage”, and “Gender” are exceptions.The component count is determined by the triples present in each dataset, so related relations can differ between WN11 and WN18.
5 Conclusion
TransG is a generative Bayesian non-parametric infinite mixture embedding model designed to address multiple relation semantics in knowledge graphs. The paper reports substantial improvements against state-of-the-art baselines and provides related materials online.
- TransG addresses multiple relation semantics by automatically discovering latent relation meanings and using a mixture of relation components for embedding.The model is described as generative, Bayesian non-parametric, and infinite mixture-based.
- The authors report substantial improvements over state-of-the-art baselines in extensive experiments.
- Related posters, slides, datasets, and code were published online.
6 Code Tricks
The implementation uses the experimental TransG class with fixed variance, CRP-based mixture assignment, and SGD-based parameter learning. Training is limited to 2,000 rounds despite slow convergence observed near 10,000 epochs.
- The experimental implementation uses “TransG,” not “TransG Hierarchical,” and fixes variance σ as a constant for numerical stabilization.
- New clusters receive the CRP factor as their mixture factor but use a random center instead of t −h because the theoretical center can be far from ground truth on big data.
- Parameter π and σ are learned with stochastic gradient ascent for efficiency rather than likelihood counting, which is theoretically natural for CRP but inefficient.
- 2,000 rounds replace training to convergence, which previously required almost 10,000 epochs because of the model’s slow convergence rate.