Source-linked AI summary
Lifted Rule Injection for Relation Embeddings
Thomas Demeester, Tim Rocktäschel, Sebastian Riedel
TL;DR
Representation-learning methods need an efficient way to incorporate commonsense rules because propositionalized first-order logic does not scale to large entity domains. The paper introduces lifted rule injection, which constrains relation embeddings and entity-tuple representations, and reports improved knowledge base completion with little computational overhead.
Problem
Propositionalizing first-order commonsense rules does not scale beyond domains with few entity tuples and rules, limiting efficient rule integration into representation-learning models.
Method
Lifted rule injection avoids grounding implication rules by imposing partial-order constraints on relation embeddings together with a non-negative entity-tuple embedding space.
Results
The model improves over a matrix factorization baseline on knowledge base completion when incorporating WordNet-derived rules.
Takeaways & Limitations
The approach provides a way to leverage external commonsense knowledge for inferring facts, especially when annotation is costly and training data are limited.
Takeaways & Limitations
The presented approach handles implications but not general first-order rules, including conjunctions, disjunctions, and negations.
Abstract
from arXiv · showhide
Methods based on representation learning currently hold the state-of-the-art in many natural language processing and knowledge base inference tasks. Yet, a major challenge is how to efficiently incorporate commonsense knowledge into such models. A recent approach regularizes relation and entity representations by propositionalization of first-order logic rules. However, propositionalization does not scale beyond domains with only few entities and rules. In this paper we present a highly efficient method for incorporating implication rules into distributed representations for automated knowledge base construction. We map entity-tuple embeddings into an approximately Boolean space and encourage a partial ordering over relation embeddings based on implication rules mined from WordNet. Surprisingly, we find that the strong restriction of the entity-tuple embedding space does not hurt the expressiveness of the model and even acts as a regularizer that improves generalization. By incorporating few commonsense rules, we achieve an increase of 2 percentage points mean average precision over a matrix factorization baseline, while observing a negligible increase in runtime.
1 Introduction
The paper introduces lifted rule injection to incorporate implication-based commonsense knowledge without propositionalizing rules over every entity tuple. It constrains relation and entity-tuple representations so injected implications remain computationally efficient and support generalization.
- 1 Introduction: Propositionalizing first-order rules creates too many loss terms for large databases, limiting the scalability of earlier rule-regularization methods.The limitation arises because rules are expanded over observed entity tuples rather than handled at the relational level.
- 1 Introduction: Lifted rule injection incorporates first-order implication rules while modeling only training facts, avoiding a separate loss term for every grounded rule.The method minimizes an upper bound that depends on rule-mentioned relation representations and a rule-independent constraint on entity-tuple embeddings.
- 1 Introduction: The method enforces implications by partially ordering relation embeddings, such as requiring each isMan component to be no larger than the corresponding isMortal component.Under non-negative entity-tuple representations, this ordering guarantees the implication for any entity tuple.
- 1 Introduction: The non-negative or approximately Boolean entity-tuple space unexpectedly improves generalization rather than reducing the model’s expressiveness.The paper identifies this restriction as a beneficial regularization effect and reports more structured relation representations.
- 1 Introduction: The approach incorporates commonsense knowledge from resources such as WordNet and improves knowledge base completion while retaining computational efficiency.The paper’s listed contributions include WordNet rule injection, improved completion performance, and qualitative evidence that implication constraints structure relation embeddings.
2 Background
The paper uses matrix factorization to represent relations and entity tuples as latent vectors, scoring facts by vector compatibility. Training ranks observed facts above sampled unobserved facts with a BPR-style reconstruction objective and L2 regularization.
- 2 Background: Matrix factorization represents each relation and entity tuple with a k-dimensional latent vector, and combines them into relation-instance facts.A fact is written as ⟨r,t⟩, with O denoting the observed training facts.
- 2 Background: The model scores relation–tuple compatibility with the dot product r⊤t, assigning higher scores to valid facts than to negative ones.Because negative evidence is typically unavailable, training uses a Bayesian Personalized Ranking objective.
- 2 Background: Each positive training fact is compared with a randomly sampled unobserved fact for the same relation.The reconstruction objective therefore learns to rank observed valid facts above unobserved facts.
- 2 Background: L2 regularization is added to relation and entity-tuple embeddings to reduce overfitting, with α controlling the regularization strength.The overall objective combines reconstruction loss with this embedding penalty.
3 Lifted Injection of Implications
Lifted rule injection replaces entity-tuple-dependent grounding with relation-level constraints, using non-negative tuple embeddings to impose implications efficiently. The resulting loss is independent of the number of entity tuples and preserves implication validity beyond observed training facts.
- 3.1 Grounded Loss Formulation: Grounded implication losses remain computationally expensive and may not generalize from training facts to unseen inferred facts.The lifted formulation addresses both limitations by avoiding explicit grounding.
- 3.3 Approximately Boolean Entity Tuples: Non-negative tuple embeddings enable a tuple-independent upper bound, while approximately Boolean sigmoid embeddings implement the required restriction in practice.The model restricts tuple representations to the hypercube [0, 1]^k.
- 3.2 Lifted Loss Formulation: The lifted loss replaces per-tuple implication terms with relation-level constraints, making rule injection independent of the number of entity tuples.It encourages component-wise ordering rp ≤ rq between relation embeddings.
- 3.4 Convex Implication Loss: A convex hinge-style implication loss prevents continued separation after the ordering is satisfied, unlike the logistic reconstruction loss.A small margin δ ensures gradients persist until the inequality is met.
- 3.5 Efficiency: Evaluating one lifted implication is comparable to evaluating one pair of training facts, so its extra runtime is negligible when rules are fewer than facts.This is the main computational advantage over grounded rule injection.
4 Related Work
The paper situates lifted rule injection among methods combining symbolic rules with distributed representations and distinguishes it through scalability, generalization, and flexible training. Its approach is narrower than prior first-order logic frameworks because it focuses on implications.
- Broader Connections: Relation-level partial ordering can transfer rule information from surface patterns to structured Freebase relation predictions.The paper presents this as an interaction beyond direct relations.
- Scope: The model is less general than Rocktäschel et al.’s framework because it addresses implications rather than a wider range of first-order logic rules.Extending lifted rule injection beyond implications is identified as future work.
- Advantages over Related Work: Lifted rule injection scales with the number of implication rules rather than rules multiplied by observed facts.This contrasts with grounded approaches that create contributions for rule instances over training facts.
- Advantages over Related Work: Injected implications remain valid for unseen facts because validity depends on the imposed ordering of relation representations.Grounded training on observed facts does not provide the same guarantee.
- Advantages over Related Work: The method supports varied loss functions, including the rank-based loss used by Riedel et al., unlike Rocktäschel et al.’s model.The paper links this flexibility to improved accuracy in a zero-shot experiment.
- Independence Assumption: Unlike prior propositionalized rule models, this approach does not assume independence between the atoms in an implication.Equivalent relations reduce its constraints to equality rather than relying on independent atom probabilities.
5 Experiments and Results
Experiments show that restricting tuple embeddings improves generalization, lifted rule injection supports zero-shot inference efficiently, and WordNet rules improve relation extraction while preserving implication asymmetry.
- 5.1 Restricted Embedding Space: Restricting tuple embeddings to an approximately Boolean space does not reduce expressiveness and slightly improves generalization over real-valued embeddings.The authors attribute the improvement to a regularization effect, with restricted embeddings producing more cluster-like relation representations.
- 5.2 Zero-shot Learning: FSL surpasses R15-Joint and matrix factorization once more than 7.5% of Freebase training facts are available, while using a faster compatible objective.FSL starts below R15-Joint with no Freebase training facts but makes better use of rules and limited data beyond the 7.5% threshold.
- 5.3 Injecting Knowledge from WordNet: A 2% weighted MAP increase over model FS and 4% over matrix factorization F follows from injecting 36 high-quality WordNet rules.The evaluation measures only indirect effects because no injected rule directly implies a test relation.
- 5.3 Injecting Knowledge from WordNet: WordNet rules polarize relation embeddings and improve predictions for structured Freebase relations through indirect effects beyond the explicitly constrained relations.The embedding visualization compares model F without rules against FSL with rules; the paper gives parent/child prediction as an example.
- 5.6 Asymmetric Character of Implications: FSL preserves asymmetric implication behavior, whereas the unrestricted model mainly captures relation similarity and can favor implications in the wrong direction.Injected rules distinguish actual implications from inverse inferences, although equivalent relations can still yield high inverse scores.
6 Conclusions
Lifted rule injection incorporates first-order implication rules without costly grounding, while its constrained entity-tuple space appears to improve regularization rather than reduce expressiveness. WordNet-derived rules improve knowledge base completion over a matrix factorization baseline and may help when labeled facts are scarce.
- The method avoids costly rule grounding and remains independent of the entity-domain size.Rules are satisfied for observed and unobserved facts.
- Restricting entity-tuple embeddings does not impair expressiveness and appears to provide beneficial regularization.
- WordNet-derived rules improve knowledge base completion over a matrix factorization baseline.
- The approach is especially useful where annotation is costly and only small amounts of training facts are available.
- Future work must extend the approach beyond implications to conjunctions, disjunctions, negations, and neural methods beyond matrix factorization.