Source-linked AI summary
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji
TL;DR
The paper addresses how to learn expressive entity and relation representations from complex, incomplete knowledge graphs. HittER uses hierarchical Transformers and masked entity prediction to combine local relational context with source-entity information, achieving state-of-the-art link prediction results and supporting BERT-based factoid question answering.
Problem
Learning useful knowledge graph representations is difficult because single entity vectors omit relational context, while prior context-aware methods are limited by shallow architectures.
Method
HittER uses a hierarchical Transformer whose bottom block encodes entity-relation pairs and whose top block aggregates a source entity’s graph neighborhood, with masked entity prediction balancing contextual and source-entity information.
Results
HittER achieves new state-of-the-art results on FB15K-237 and WN18RR and its learned representations can be used by BERT for factoid question answering.
Takeaways & Limitations
Relational context from local neighborhoods can improve knowledge graph embeddings and can be integrated into Transformer-based language models for knowledge-intensive tasks.
Takeaways & Limitations
HittER currently aggregates only local graph neighborhoods, and broader graph context remains an avenue for potential improvement.
Abstract
from arXiv · showhide
This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.
1 Introduction
Knowledge graph embeddings support applications but struggle to encode all information about an entity in one vector. HittER addresses this by combining deep hierarchical Transformer contextualization with entity representations, achieving state-of-the-art link prediction results on two benchmarks.
- Motivation: Knowledge graph representation learning matters because knowledge graphs support natural language understanding, reasoning, and knowledge graph completion.The paper frames link prediction as a proxy for evaluating representation quality in incomplete graphs.
- Motivation: Traditional KGE methods encode entities as single vectors, making it difficult to retrieve relational information available in an entity’s graph context.For example, predicting Sunnyvale’s county can use neighboring information such as its connection to California.
- Prior limitations: Earlier context-aware GNN and attention methods are restricted in expressiveness by their shallow architectures.The introduction identifies shallow network depth as a limitation of these approaches.
- Approach: HittER jointly learns entity and relation representations by aggregating information from graph neighborhoods with a deep hierarchical Transformer.Its bottom block provides relation-dependent neighborhood entity embeddings, while its top block aggregates graph-context information.
- Results: New state-of-the-art results on FB15K-237 and WN18RR show HittER’s effectiveness for link prediction.Link prediction evaluates whether learned representations generalize relational knowledge to unseen facts.
2 HittER
HittER uses hierarchical Transformers to encode entity-relation pairs and aggregate a source entity’s relational neighborhood. Masking and source-entity recovery balance contextual information against the source entity’s own information during training.
- 2.1 Transformers for Link Prediction: The model represents link prediction as scoring candidate entities for an incomplete triplet and training with cross-entropy.A Transformer [CLS] representation is compared with candidate entity embeddings to produce plausibility scores.
- 2.2 Hierarchical Transformers: HittER’s bottom Transformer encodes entity-relation pairs, while its top Transformer aggregates their structural features into contextual information for link prediction.The bottom block compresses each pair into one vector, reducing the inputs passed to the top block and avoiding long sequences.
- 2.3 Balanced Contextualization: HittER uses source-entity masking or random replacement so the model learns to use relational context rather than ignoring it.The masking probability is dataset-specific and can be increased when the neighborhood is denser.
- 2.3 Balanced Contextualization: An auxiliary masked entity prediction task recovers the perturbed source entity from contextual information, preserving the source entity’s contribution when context is insufficient.The source-entity output embedding predicts the correct entity through an additional classification loss.
- 2.3 Balanced Contextualization: Neighborhood sampling regularizes training, while removing the ground-truth target from the neighborhood prevents train-test mismatch.Only a fraction of neighboring entities appears in each training example.
3 Link Prediction Experiments
HittER is evaluated for link prediction on FB15K-237 and WN18RR using standard ranking metrics, with experiments examining overall performance and the contribution of contextualization. It outperforms prior methods broadly, while balanced context is especially beneficial on sparse, noisy graphs and for multi-hop or difficult relations.
- 3.4 Experimental Results: HittER outperforms previous methods by a substantial margin across nearly all metrics on the two standard datasets.The comparison excludes models using excessively large embedding sizes.
- 3.5 Ablation Studies: Balanced contextualization significantly improves MRR on both datasets, whereas adding context without balancing does not benefit the model.The effect is especially strong on WN18RR, whose graph is sparser and noisier.
- 3.5 Ablation Studies: Contextual information substantially improves performance on WN18RR hypernym and member meronym relations, which include many challenging one-to-many examples.The relation-level comparison reports dev MRR and relative improvement for models with or without the context Transformer.
- 3.5 Ablation Studies: Contextual HittER performs much better for examples separated by two to four graph hops, but its improvement shrinks as hop count increases.MRR decreases as the number of hops grows, indicating that longer paths are harder to aggregate meaningfully.
4 Factoid QA Experiments
The experiments integrate HittER representations into BERT through cross-attention and evaluate the combination on two Freebase factoid QA datasets. HittER significantly improves BERT’s question-answering ability, particularly for questions linked to entities in HittER’s training graph.
- The integration adds a cross-attention module after each BERT self-attention module, using HittER outputs as keys and values.Queries come from the previous BERT layer, allowing HittER knowledge information to flow into BERT.
- The study evaluates HittER-BERT on FreebaseQA and WebQuestionSP, whose questions provide a context entity and inferred relation for constructing HittER inputs.
- Table 6 reports QA accuracy averages and standard deviations across five random runs for the two Freebase-based datasets.
- Because FB15K-237 covers only a small portion of Freebase, the experiments also report a filtered setting retaining examples whose context and answer entities occur in FB15K-237.
- HittER’s representation significantly enhances BERT’s question answering ability, especially for questions related to entities in its training graph.The reported results are from two Freebase-based factoid QA datasets.
5 Related Work
Prior knowledge graph embedding work models entities and relations with geometric transformations, neural interaction modules, or graph-context methods. HittER differs by using Transformers to capture both entity-relation and entity-context interactions.
- The paper limits its scope to KGE methods relying purely on entities and relations without external resources.
- Earlier KGE methods use translational, bilinear, rotational, convolutional, and other geometric or neural transformations to model entity-relation interactions.
- Graph neural approaches exploit graph context, including relational graph convolution and edge-attention mechanisms for multi-relational knowledge graphs.
- HittER uses an advanced Transformer rather than a simple single-layer attention formulation to capture both entity-relation and entity-context interactions.
6 Conclusion and Future Work
The paper concludes that HittER learns effective knowledge graph representations from local neighborhoods, improves link prediction, and can support BERT on factoid QA. Future work includes broader graph context and larger-scale knowledge graph training.
- HittER outperforms previous approaches on link prediction, achieving new state-of-the-art results on FB15K-237 and WN18RR.The conclusion attributes this result to contextual information from a local neighborhood.
- HittER’s learned knowledge representation can be used by BERT to answer factoid questions.
- The model currently aggregates contextual information only from a local graph neighborhood, motivating extensions to broader graph context for potential improvements.
- The authors plan to train HittER on large-scale knowledge graphs so more NLP models can benefit in knowledge-intensive tasks.
A Embedding Clustering
The embedding-clustering analysis shows that learned entity representations often group entities by type in FB15K-237, while WN18RR clusters are looser but remain topic-relevant.
- FB15K-237 clusters usually contain entities of the same type, whereas WN18RR clusters are looser but remain relevant to the central word’s topic.Examples include countries, government systems, and American voice actresses in FB15K-237.
B Factoid QA Experiment Details
The QA integration adds cross-attention between BERT and corresponding HittER layers. The models differ in layer count and dimensionality, and QA fine-tuning uses separate learning rates for pretrained and newly added parameters.
- Cross-attention connects each BERT layer to the corresponding HittER output, using previous-layer BERT queries and HittER keys and values.
- The pretrained BERT and HittER models differ in their numbers of layers and dimensionality.
- QA models are fine-tuned for 20 epochs with batch size 16, using learning rates of 5e−6 for pretrained weights and 5e−5 for new cross-attention modules.
C.1 Right Context for Link Prediction
The paper studies structural context beyond individual entities, focusing on local neighborhoods and relational information. Results indicate that a simple local neighborhood can substantially improve link prediction, while evidence for graph paths remains unclear.
- C.1 Right Context for Link Prediction: Structural graph information can be represented through paths, subgraphs, or local neighborhoods, using relation types, entities, or both.
- C.1 Right Context for Link Prediction: A simple local neighborhood is sufficient to greatly improve a link prediction model.
- C.1 Right Context for Link Prediction: Masking the source entity throughout early FB15K-237 experiments does not substantially harm performance, suggesting dense contextual information can replace it.
- C.1 Right Context for Link Prediction: Graph paths and local neighborhoods have been proposed jointly, but there is still no clear evidence that graph paths improve knowledge graph embeddings.
C.2 Limitations of the 1vsAll Scoring
HittER uses 1vsAll scoring for efficient training and inference, but this choice sacrifices three-way interactions between source, relation, and target entities. The paper identifies early source-target fusion as a promising direction for future work.
- C.2 Limitations of the 1vsAll Scoring: 1vsAll scoring supports fast training and inference but loses three-way interactions among source entity, predicate, and target entity.
- C.2 Limitations of the 1vsAll Scoring: The scoring paradigm represents an incomplete triplet and computes its dot product with every candidate target entity embedding.
- C.2 Limitations of the 1vsAll Scoring: Early source-target interactions can provide valuable information for representing incomplete triplets, but incorporating them effectively and efficiently remains future work.