PaperlayerView exact revision on arXiv

Source-linked AI summary

Type-Constrained Representation Learning in Knowledge Graphs

Denis Krompaß, Stephan Baier, Volker Tresp

arXiv:1508.02593v2cs.AIcs.LG

TL;DR

Knowledge-graph latent-variable models need semantic prior knowledge to model incomplete and error-prone graphs effectively. The paper integrates type-constraints into several state-of-the-art models and evaluates an observation-based alternative when constraints are missing or fuzzy. Type-constraints significantly improve link-prediction quality, with gains up to 77%, while the local closed-world assumption also helps but does not generally replace schema-derived constraints.

  • Problem

    Knowledge graphs are incomplete and error-prone, motivating methods that use their schema-level semantic information when modeling and completing them.

  • Method

    The authors integrate schema-derived type-constraints into RESCAL, TransE, and mwNN, and evaluate a local closed-world assumption that approximates constraints from observed triples.

  • Results

    Type-constraints significantly improve link-prediction quality across TransE, RESCAL, and mwNN, with improvements up to 77%, especially at low embedding dimensionality.

  • Takeaways & Limitations

    Type-constraints are essential for latent-variable knowledge-graph models within the studied setting, while LCWA is useful when constraints are absent or fuzzy.

  • Takeaways & Limitations

    The local closed-world assumption does not generally replace extracted type-constraints, whose relative advantage varies across datasets and models.

Abstract

from arXiv · show

Large knowledge graphs increasingly add value to various applications that require machines to recognize and understand queries and their semantics, as in search or question answering systems. Latent variable models have increasingly gained attention for the statistical modeling of knowledge graphs, showing promising results in tasks related to knowledge graph completion and cleaning. Besides storing facts about the world, schema-based knowledge graphs are backed by rich semantic descriptions of entities and relation-types that allow machines to understand the notion of things and their semantic relationships. In this work, we study how type-constraints can generally support the statistical modeling with latent variable models. More precisely, we integrated prior knowledge in form of type-constraints in various state of the art latent variable approaches. Our experimental results show that prior knowledge on relation-types significantly improves these models up to 77% in link-prediction tasks. The achieved improvements are especially prominent when a low model complexity is enforced, a crucial requirement when these models are applied to very large datasets. Unfortunately, type-constraints are neither always available nor always complete e.g., they can become fuzzy when entities lack proper typing. We show that in these cases, it can be beneficial to apply a local closed-world assumption that approximates the semantics of relation-types based on observations made in the data.

1 Introduction

Knowledge graphs support applications such as web search and question answering but remain incomplete and error-prone. This work evaluates type-constraints and local closed-world assumptions as prior knowledge for latent-variable models of knowledge graphs.

  • Knowledge graphs have become valuable information sources for applications including web search and question answering.
  • Despite containing billions of facts, knowledge graphs remain incomplete and are not error-free.Examples include missing birthplaces for many persons and missing descriptions of what scientists are known for.
  • Latent-variable representation-learning methods learn entity and relation-type embeddings for knowledge-graph completion, cleaning, and construction.
  • Schema-based knowledge graphs provide type-constraints that encode semantic roles for relation-types, such as restricting marriedTo to Person instances.
  • The study integrates type-constraints into RESCAL, TransE, and a multiway neural network to assess their general value across diverse latent-variable models.
  • Because type-constraints may be incomplete when entities lack proper typing, the study also evaluates a local closed-world assumption based on observed triples.

2 Latent Variable Models for Knowledge Graph Modeling

The paper reviews three diverse latent-variable approaches for knowledge-graph modeling: tensor factorization, translational embeddings, and a multiway neural network. These models learn latent representations and confidence scores for triples using distinct objectives and training procedures.

  • RESCAL, TransE, and mwNN were selected because they scale to large knowledge graphs and represent diverse modeling strategies.RESCAL is bilinear, TransE uses linear translations, and mwNN models nonlinear interactions.
  • RESCAL: RESCAL factorizes a three-way adjacency tensor into shared entity embeddings and relation-specific matrices.Its embeddings are learned by minimizing a regularized least-squares objective, efficiently optimized with alternating least-squares.
  • RESCAL: RESCAL computes triple confidence by reconstructing the subject–relation–object interaction from latent entity and relation representations.
  • TransE: TransE models a true relation as a translation from the subject embedding toward the object embedding.Confidence is based on L1 or L2 distance, and max-margin ranking is trained with stochastic gradient descent using corrupted triples.
  • Knowledge Vault Neural Network: The multiway neural network predicts triple confidence from stacked subject, predicate, and object embeddings passed through nonlinear layers.Its weights are trained with a Bernoulli cost function using stochastic gradient descent; corrupted triples provide negative evidence.

3 Prior Knowledge On Relation-Type Semantics

The paper incorporates relation-type semantics into latent-variable KG models by restricting model components and corrupted triples to type-compatible entities. When curated constraints are incomplete or inconsistent, a local closed-world assumption approximates them from observed triples.

  • Type-constraints: Type-constraints encode domain and range classes for relation-types, specifying which subject and object entity types a relation should connect.RDFS concepts provide these constraints; domain applies to subjects and range to objects.
  • Type-constrained RESCAL: RESCAL restricts each relation’s latent embeddings and modeled subgraph to entities satisfying its domain and range constraints.The constrained adjacency matrix and indexed entity embeddings are used in ALS optimization.
  • Type-constrained SGD: TransE and mwNN use type-compatible entity subsets when corrupting triples during SGD training, preventing generated negatives from violating relation semantics.TransE restricts both corrupted subjects and objects; mwNN restricts corrupted objects to the predicate’s range.
  • Constraint quality: Incomplete or inconsistent typing can make curated constraints fuzzy, risking exclusion of true triples or inclusion of semantically meaningless triples.Applying constraints directly is unsafe for relations with frequent inconsistencies, while missing constraints leave many triples semantically unconstrained.
  • Local closed-world assumption: The local closed-world assumption defines a relation’s domain and range from entities observed in subject and object positions for that relation.It operates at the instance level rather than the class level and relies solely on observed triples.

4 Experimental Setup

The experiments compare three representative latent-variable models across diverse KG datasets under low-complexity settings. Link prediction is evaluated using held-out triples, with datasets designed to represent general-purpose, domain-specific, and high-quality knowledge graphs.

  • Models and settings: The study compares RESCAL, TransE, and mwNN under two settings: curated schema-derived type-constraints and a local closed-world assumption.The models are selected to represent diverse latent-variable approaches, and both settings test the value of relation-type semantics.
  • Models and settings: The experiments enforce low-dimensional latent embeddings to simulate applications where high-dimensional representations are impractical on very large datasets.The setup treats low model complexity as a central experimental condition.
  • Datasets: Datasets extracted from Freebase, DBpedia, and YAGO represent general-purpose, domain-specific, and high-quality knowledge graphs.Freebase-150k represents a general-purpose KG, DBpedia-Music a domain-specific KG, and YAGOc-195k a high-quality KG.
  • Datasets: Freebase-150k was constructed from materialized Freebase data by extracting entity types, type-constraints, and triples involving highly connected entities and sufficiently frequent relation-types.Triples with incomplete type-constraints or relation-types occurring in fewer than 100 triples were discarded.
  • Datasets: DBpedia-Music uses triples and types from 15 pre-selected object-properties covering the music domain.The selected properties include musical and genre-related relations, with genre restricted to entities covered by the other properties.
  • Datasets: YAGOc-195k uses YAGO-core entities and relation-types above specified connectivity thresholds, retaining entities sharing types used in domain and range constraints.The extraction targets YAGO’s automatically generated high-quality KG data and clean taxonomy.
  • Evaluation: Link prediction removes triples and evaluates whether models can re-predict them using 70% training, 10% validation, and 20% holdout data.Negative triples were sampled for validation and holdout evaluation, while validation supported hyper-parameter tuning.

5 Experimental Results

Across RESCAL, TransE, and mwNN, incorporating relation-type prior knowledge generally improves link-prediction quality, with the largest gains often appearing at low model complexity. The LCWA also helps substantially, but does not consistently replace schema-derived type-constraints.

  • Experimental setup: The experiments compare RESCAL, TransE, and mwNN across Freebase-150k, DBpedia-Music, and YAGOc-195 datasets using relation-type prior knowledge.Tables report AUPRC and AUROC for models using no prior knowledge, type-constraints, or the LCWA.
  • Type-Constraints are Essential: Type-constraints significantly improve link-prediction quality across all models and settings in both AUPRC and AUROC.For RESCAL on Freebase-150k, AUPRC rises from 0.327 to 0.521 at d = 10 and from 0.514 to 0.654 at d = 100.
  • Type-Constraints are Essential: 77%: mwNN’s Freebase-150k AUPRC increases from 0.437 to 0.775 at d = 10 when type-constraints are used.Type-constraints also improve mwNN on DBpedia-Music and YAGOc-195k.
  • Type-Constraints are Essential: The largest improvements most often occur at d = 10, where type-constrained models can outperform more complex models that ignore type-constraints.On Freebase-150k, mwNN reaches 0.775 AUPRC with d = 10 using type-constraints, versus 0.512 with d = 100 without them.
  • Local Closed-World Assumption – Simple but Powerful: The LCWA produces similarly large link-prediction gains, especially at d = 10, including RESCAL’s Freebase-150k AUPRC increase from 0.327 to 0.579.Other examples include TransE rising from 0.715 to 0.806 at d = 50 and mwNN rising from 0.600 to 0.714 on YAGO.
  • Local Closed-World Assumption – Simple but Powerful: The LCWA does not generally replace schema-derived type-constraints: type-constraints are clearly superior on YAGOc-195k, while the preferred approach varies elsewhere by model and embedding length.RESCAL favors the LCWA on Freebase-150k and DBpedia-Music, mwNN favors type-constraints, and longer TransE embeddings favor the LCWA.

6 Related Work

Prior work extended latent-variable knowledge-graph models with relation-type constraints, while this paper examines their broader integration beyond RESCAL. Model choice also reflects scalability considerations for large datasets.

  • The study excludes neural tensor networks because prior work found they do not scale to larger datasets.
  • Instead, the paper uses a less complex, more scalable neural network model reported to achieve comparable results to neural tensor networks.
  • Domain and range constraints had been used in RESCAL but not broadly integrated into other latent-variable methods.
  • Combining latent-variable methods with graph-feature models has been reported to increase prediction quality and decrease model complexity.

7 Conclusions and Future Work

The paper finds that semantic relation-type information improves latent-variable knowledge-graph models, especially under low embedding dimensionality. When curated type-constraints are absent or fuzzy, LCWA provides an observation-based alternative, and future work will explore further extensions and model differences.

  • 7 Conclusions and Future Work: Up to 77%, type-constraints significantly improve link-prediction quality for TransE, RESCAL, and mwNN.
  • 7 Conclusions and Future Work: Type-constraint benefits are especially prominent when embedding dimensionality—and thus model complexity—must remain very low.
  • 7 Conclusions and Future Work: LCWA approximates relation-type domain and range constraints at the instance level from observed triples when type-constraints are absent or fuzzy.
  • 7 Conclusions and Future Work: In the authors’ proposed real-world setting, curated type-constraints would be used when available and LCWA where they are absent or fuzzy.
  • 7 Conclusions and Future Work: Future work will investigate extensions combining graph-feature models with type-constraints or LCWA and compare which model differences most benefit link prediction.
Loading 1508.02593v2…