Source-linked AI summary
Neural Regression with Embeddings for Numerical Attribute Prediction in Knowledge Graphs
Rupesh Sapkota, Louis Mozart Kamdem Teyou, Moshood Yekini, Caglar Demir, Axel-Cyrille Ngonga Ngomo
TL;DR
Many transductive KGE models neglect numerical attributes, despite their presence in knowledge graphs. The paper introduces LitEm and a co-training framework to predict and incorporate these attributes, achieving best-or-second-best results across the evaluated datasets while improving link prediction mainly for bilinear models. The approach remains limited to numerical literals and relatively small graphs and embedding dimensions.
Problem
Many transductive KGE models neglect numerical attributes in knowledge graphs, limiting their representation of diverse real-world information.
Method
LitEm predicts numerical attributes from pretrained entity embeddings, while co-training jointly trains KGE models with LitEm using dynamically weighted KGE and literal losses.
Results
LitEm or LitEmbase achieves the best or second-best MAE on 8 of 11 FB15K-237 attributes, all 7 YAGO15K attributes, all 9 DB15K attributes, and all 4 Mutagenesis attributes.
Takeaways & Limitations
Co-training enables transductive KGE models to predict numerical attributes and improves link prediction performance mainly for bilinear models.
Takeaways & Limitations
The approach covers only numerical literals and was evaluated on graphs of 12,000–15,000 entities with embeddings of at most 64 dimensions.
Abstract
from arXiv · showhide
In recent years, transductive knowledge graph embedding models have been applied to tasks such as link prediction and query answering. Although knowledge graphs often contain rich numerical attributes, most embedding models neglect them, limiting their ability to represent real-world knowledge graphs with diverse information. In this work, we propose a neural regression model (LitEm) that enables transductive knowledge graph embedding models to predict numerical attributes within knowledge graphs. Experimental results demonstrate that LitEm achieves the best or second-best results on most attributes across FB15K-237, YAGO15K, DB15K, and Mutagenesis. Furthermore, we propose a co-training framework that jointly trains state-of-the-art transductive knowledge graph embedding models with LitEm, which improves link prediction performance mainly for bilinear models and simultaneously enables them to predict numerical attributes. In addition, the literal-awareness evaluation demonstrates that co-training helps models to encode and exploit attribute information in a "literal-aware'' manner, suggesting that the observed gains are not merely due to additional parameters. We publicly release our implementation at https://github.com/dice-group/dice-embeddings.
1 Introduction
The paper addresses numerical attributes overlooked by many KGE models by introducing LitEm and a co-training framework for prediction and augmentation. It also proposes dynamic loss weighting and benchmarks DB15K and Mutagenesis for numerical attribute prediction.
- Motivation: Many KGE models use relational information while overlooking numerical literals that can enrich entity representations.Numerical literals have also been used for link prediction and complex query answering.
- Contributions: LitEm is a neural regression model that predicts numerical attributes from pretrained embeddings.
- Contributions: The proposed co-training framework jointly trains KGE models with LitEm to predict and augment numerical attributes.
- Contributions: A gradient-based dynamic weighting mechanism balances KGE and literal losses during co-training.
- Contributions: DB15K and Mutagenesis are repurposed as benchmarks for numerical literal prediction, with first reported results on both.
2 Background
Knowledge graphs represent entities, relationships, and numerical attribute values as structured triples. KGE models embed entities and relations for downstream tasks, while literal embedding models learn representations for numerical attributes.
- Knowledge Graphs: A knowledge graph contains relational triples and numerical attribute triples linking entities to attribute values.The formal representation separates entities, relations, attributes, and real-valued numerical values.
- Transductive Knowledge Graph Embeddings: KGE models represent entities and relations in vector spaces and use parameterized scoring functions for tasks such as link prediction.Examples include translation, multiplicative, complex-valued, rotational, quaternion, octonion, dual-quaternion, and Clifford-algebra models.
- Literal Embedding Models: Literal embedding models extend KGE approaches by learning embeddings for numerical attributes and capturing real-valued entity properties.Most existing approaches do not jointly learn attribute representations and predict numerical values using transductive KGE models.
- Literal Embedding Models: A literal embedding model scores an entity-attribute pair to produce a predicted numerical value.Its parameters typically include attribute embeddings with dimensionality d_a.
3 Related Work
Prior work includes graph-based, regression, propagation, and language-model approaches for numerical attribute prediction. LitEm instead decodes attributes from existing embeddings and combines this decoder with model-agnostic co-training.
- Numerical Attribute Prediction: Earlier numerical attribute prediction methods jointly learned relational and attribute networks, used regression or propagation baselines, or propagated values through multi-relational neighborhoods.
- LitEm: LitEm regresses from frozen entity embeddings through a gated-residual network without adding an auxiliary objective to the KGE model.
- Comparison: Unlike LiteralE, which gates entity embeddings during KGE training, LitEm gates attribute prediction from embeddings that already exist.
- Co-training: LitEm pairs its decoder with model-agnostic co-training that updates embeddings through the literal loss alone.
4 Literal Embedding Model
LitEm predicts numerical attributes from entity and attribute embeddings using a residual regression network with optional gating. Its co-training framework pairs relational and attribute triples and balances their losses while updating the shared embeddings.
- 4 Literal Embedding Model: LitEm takes pretrained entity embeddings and attribute embeddings as inputs to regress an entity’s numerical attribute value.Entity embeddings are fixed by default, while attribute embeddings are trainable.
- 4 Literal Embedding Model: The base model concatenates entity and attribute embeddings, applies a ReLU-transformed linear layer, adds a residual connection, and projects the result to a scalar.The second linear layer produces the predicted numerical value.
- 4 Literal Embedding Model: The gated residual mechanism splits a transformed representation into a value and gate, then modulates the value through element-wise multiplication with a sigmoid gate.This mechanism controls the contribution of the residual pathway.
- 4 Literal Embedding Model: Co-training jointly processes relational triples with a KGE model and numerical attribute triples with LitEm, injecting attribute information into shared entity embeddings.Standalone LitEm keeps entity embeddings fixed, whereas co-training updates them through the combined training objective.
- 4 Literal Embedding Model: The combined objective uses a dynamically scaled LitEm loss, with λ capped at 0.5 so the structural objective retains at least half the weight.The scaling factor increases when structural gradients dominate and decreases when literal gradients are larger.
5 Evaluation
Evaluation shows LitEm predicts numerical attributes competitively across four datasets and that co-training affects link prediction differently across embedding families. Literal-awareness tests further indicate that co-trained models use numerical values rather than merely receiving additional parameters.
- Literal prediction: LitEm or LitEmbase ranks best or second-best on 8 of 11 FB15K-237 attributes and all 7 YAGO15K attributes.On FB15K-237, LitEm outperforms KGE-Reg on every attribute and LR on most attributes; on YAGO15K, both variants achieve the strongest performance across attributes.
- Embedding-model comparison: TransE and RotatE generally achieve lower literal prediction loss than multiplicative-scoring models, with TransE best on most attributes.The results suggest translational geometries make numerical attributes easier to recover, while complex scoring functions may require scoring-function-aware decoders.
- Additional benchmarks: LitEm variants outperform linear regression on every DB15K and Mutagenesis attribute and outperform MrAP on most attributes.Notable MrAP comparisons include longitude 11.776 vs. 16.992, height 0.346 vs. 1.403, and mutagenesis#charge 0.022 vs. 0.397.
- Additional benchmarks: LitEm or LitEmbase ranks best or second-best on all 9 DB15K and all 4 Mutagenesis attributes.DB15K and Mutagenesis are introduced as benchmarks for numerical attribute prediction, using per-attribute linear regression and MrAP as comparison baselines.
- Link prediction and co-training: Co-training improves link prediction most consistently for bilinear models, while TransE and RotatE experience small reductions attributed to gradient conflict and scoring-function constraints.Across 24 co-trained model-dataset settings, MRR ranges from 0.179 to 0.382 and Hits@1 from 0.115 to 0.299; 18 settings remain below MRR 0.300.
- Literal prediction under co-training: Co-training improves literal prediction for most attributes overall, but FB15K-237 results are mixed: date_of_birth improves in 5 of 8 models while other attributes often worsen.The reported literal-augmented-model results come from combined training rather than post-augmentation regression.
- Literal-awareness evaluation: LitEm co-trained models achieve higher synthetic-triple accuracy on original numerical values than randomized values, unlike LiteralE and ReaLitE.This evaluation supports that LitEm co-training helps models encode and exploit numerical information itself.
6 Discussion
The discussion identifies scoring-function geometry and data-split design as key factors in LitEm’s behavior. Co-training benefits bilinear models more than translational models, while attribute prediction remains transductive and is robust to moderate data reduction.
- Literal prediction: A shared attribute-conditioned regression model outperforms separate per-attribute regressors, while additional layers provide only marginal gains.This suggests cross-attribute sharing regularizes prediction, whereas regression depth is not the primary bottleneck.
- Literal prediction: TransE frozen embeddings reach mean R2 0.28 with per-attribute ridge regression, compared with 0.18 for ComplEx and DistMult.The authors propose scoring-function-aware literal decoders as future work.
- Co-training effects: Co-training significantly improves link prediction for aggregated bilinear models across 4 models, 3 datasets, and 5 random seeds.The discussion associates this concentration with bilinear models’ greater freedom to accommodate literal information while preserving structural objectives.
- Co-training effects: Translational models show little co-training benefit because their strict geometric constraints leave less freedom to incorporate literal gradients.The examples given are TransE’s h + r ≈ t and RotatE’s h ◦ r ≈ t constraints.
- Evaluation design: The disjoint attribute split prevents entities from appearing as supervised literal targets in both training and test sets, while retaining transductive entity embeddings.LitEm performs comparably to the standard split and better on some attributes, but it is not inductive because every predicted entity requires a trained embedding.
- Evaluation design: LitEm degrades gracefully at 80% and 60% training data, with visible degradation mainly at 40% for date- and coordinate-valued attributes.Most attributes remain within a few per cent ΔMAE at 80%, and accuracy is close to full-data performance at 60%.
- Limitations and future work: The approach is limited to numerical literals and experiments using embeddings of at most 64 dimensions on graphs containing 12,000–15,000 entities.Whether the patterns persist at higher dimensionality, larger scale, or with model-specific tuning remains untested.
7 Conclusion
The paper introduces LitEm for numerical attribute prediction from pre-trained knowledge-graph embeddings and evaluates it across four datasets. It also presents co-training that improves link prediction mainly for multiplicative models while enabling literal-aware representations.
- Contributions: LitEm predicts numerical attributes of knowledge-graph entities by leveraging pre-trained embeddings.
- Numerical attribute prediction: LitEm or LitEmbase achieves the best or second-best MAE on 8 of 11 FB15K-237 attributes and all 7 YAGO15K attributes.
- Numerical attribute prediction: LitEm or LitEmbase ranks best or second-best on all 9 DB15K and all 4 Mutagenesis attributes, providing the first reported results on both benchmarks.
- Co-training: The co-training framework jointly incorporates numerical attributes into embedding methods and improves link prediction mainly for models with multiplicative scoring functions.
- Co-training: Co-training simultaneously enables embedding models to predict numerical attributes and become literal-aware.