Source-linked AI summary
Adaptive Attentional Network for Few-Shot Knowledge Graph Completion
Jiawei Sheng, Shu Guo, Zhenyu Chen, Juwei Yue, Lihong Wang, Tingwen Liu, Hongbo Xu
TL;DR
Few-shot KG completion must infer missing facts for relations with limited examples, while prior methods overlook dynamic entity roles and query-dependent reference contributions. FAAN adaptively represents entities and references through attention-based encoders and aggregation, achieving new state-of-the-art results across few-shot sizes on two public datasets.
Problem
Few-shot KG completion involves long-tail relations with limited triples, while prior methods use static representations despite dynamic entity roles and reference contributions.
Method
FAAN uses an adaptive neighbor encoder for task-oriented entity roles and Transformer-based, attention-driven aggregation to adapt reference representations to queries.
Results
FAAN achieves new state-of-the-art results in few-shot link prediction on NELL and Wikidata with different few-shot sizes.
Takeaways & Limitations
Adaptive entity and reference representations provide richer, fine-grained representations for few-shot knowledge acquisition.
Takeaways & Limitations
Future work may need more advanced few-shot relation modeling and additional contextual information such as textual descriptions.
Abstract
from arXiv · showhide
Few-shot Knowledge Graph (KG) completion is a focus of current research, where each task aims at querying unseen facts of a relation given its few-shot reference entity pairs. Recent attempts solve this problem by learning static representations of entities and references, ignoring their dynamic properties, i.e., entities may exhibit diverse roles within task relations, and references may make different contributions to queries. This work proposes an adaptive attentional network for few-shot KG completion by learning adaptive entity and reference representations. Specifically, entities are modeled by an adaptive neighbor encoder to discern their task-oriented roles, while references are modeled by an adaptive query-aware aggregator to differentiate their contributions. Through the attention mechanism, both entities and references can capture their fine-grained semantic meanings, and thus render more expressive representations. This will be more predictive for knowledge acquisition in the few-shot scenario. Evaluation in link prediction on two public datasets shows that our approach achieves new state-of-the-art results with different few-shot sizes.
1 Introduction
Few-shot KG completion addresses missing facts for long-tail relations, where static entity and reference representations overlook task-dependent roles and query-dependent contributions. FAAN models these dynamic properties adaptively and reports new state-of-the-art results across few-shot sizes.
- Knowledge graphs support NLP tasks but suffer from incompleteness, motivating KG completion through inferred missing facts.
- Long-tail relations have few training triples, limiting conventional KG embedding methods on KG completion.
- Earlier few-shot methods use static neighbor weights, although entities may have different roles and references may contribute differently across queries.
- FAAN learns dynamic entity and reference representations with an adaptive neighbor encoder, Transformer blocks, and attention-based reference aggregation.
- FAAN achieves new state-of-the-art results on few-shot link prediction over NELL and Wikidata with different few-shot sizes.
2 Related Work
Related work spans general KG embedding and few-shot relational learning, but prior few-shot methods learn static entity or reference representations. FAAN instead targets dynamic representations for entities and references.
- KG embedding methods include translation-based, semantic matching, and neural network models, but traditional approaches require sufficient triples for all relations.
- Few-shot KG methods include GMatching, FSRL, and MetaR, using neighbor encoding, attention, matching, or transferred task knowledge.
- Prior few-shot relational learning studies learn static entity or reference representations and ignore their dynamic properties.
- Dynamic representations have also been studied in traditional KG completion, image captioning, and sequential recommendation, but in different application settings.
3 Background
Few-shot KG completion ranks the true tail among candidates using a small reference set for each relation. Training uses task-specific references and queries, while testing targets unseen relations with access to a background KG.
- A KG is represented as triples (h, r, t), and few-shot completion predicts a missing tail entity from (h, r, ?) using reference pairs.
- When the reference set contains K pairs and K is very small, the task is called K-shot KG completion.
- The objective is to rank the true tail above false candidate entities for each query.
- Meta-training and meta-testing use relation-specific reference/query tasks, with testing relations unseen during meta-training.
- Testing assumes access to a background KG containing relations excluded from the meta-training and meta-testing tasks.
4 Our Approach
FAAN combines adaptive entity encoding, Transformer-based entity-pair representations, and query-aware reference matching for few-shot KG completion. Its attention mechanisms adapt entity roles and reference contributions to task relations and queries.
- Framework: FAAN has three components: an adaptive neighbor encoder, a Transformer encoder for entity pairs, and an adaptive matching processor.The framework compares K-shot references with a query after constructing adaptive representations.
- Adaptive Neighbor Encoder: The adaptive neighbor encoder weights one-hop neighbors according to their relevance to the task relation, producing role-aware entity representations.Neighbor relations more related to the task relation receive higher relevance scores and contribute more to neighbor embeddings.
- Adaptive Neighbor Encoder: FAAN models task relations under a translation assumption, representing relation r from entity embeddings as r = t − h.The entity embeddings are pretrained on the background KG, and alternative embedding methods are left for future work.
- Transformer Encoder: A Transformer encodes each head–relation–tail sequence, with the task-relation hidden state serving as the entity-pair representation.The representation captures semantic roles of entities and fine-grained meanings associated with different entity pairs.
- Adaptive Matching Processor: The adaptive matching processor uses query–reference similarity to aggregate K-shot references into a query-dependent representation for prediction.References with meanings similar to the query receive greater attention, while the final similarity score is expected to be larger for true queries.
- Model Training: FAAN trains by sampling K-shot positive references, retaining remaining positives as queries, corrupting tails for negatives, and optimizing a margin-based hinge loss.Adam optimization and L2 regularization are used during meta-training.
5 Experiments
The experiments evaluate FAAN through link prediction on the NELL and Wiki1 few-shot KG completion datasets. The study compares it with conventional KG embedding methods and prior few-shot relational learning approaches.
- Evaluation Setup: FAAN is evaluated through link prediction experiments for few-shot KG completion.The experiments use two public benchmark datasets, NELL and Wiki1.
- Datasets: NELL and Wiki1 contain 67 and 183 few-shot tasks, respectively, constructed from relations with more than 50 and fewer than 500 triples.The datasets use relation splits for training, validation, and testing, with candidate entities selected using type constraints.
- Comparison Methods: The comparison includes five KG embedding baselines: TransE, DistMult, ComplEx, SimplE, and RotatE.These methods learn entity and relation embeddings by modeling relational structures and require sufficient training triples for each relation.
- Comparison Methods: The few-shot relational learning baselines include GMatching, FSRL, and MetaR.GMatching assumes equal neighbor contributions, FSRL uses fixed attention, and MetaR transfers shared knowledge from references to queries.
5.3 Implementation Details
The experiments use 5-shot KG completion with established baselines and dataset-specific model settings. FAAN is configured with fixed neighbor sampling and Transformer architectures for NELL and Wiki.
- The evaluation uses 5-shot KG completion tasks and compares implemented baselines with reported MetaR results.MetaR results are reported directly to avoid re-implementation bias.
- Entity neighbors are randomly sampled and fixed before training, with a maximum of 50 neighbors on both datasets.
- Embedding dimensionality is set to 50 for NELL and 100 for Wiki.
- FAAN uses 3 and 4 Transformer layers, with 4 and 8 attention heads, for NELL and Wiki, respectively.
5.4 Evaluation Metrics
Performance is evaluated by ranking the correct tail entity among candidate substitutions using MRR and Hits@N. On NELL and Wiki, FAAN outperforms traditional KG embedding methods and few-shot baselines, including MetaR.
- Test triples are evaluated by ranking the correct entity among all candidate tail substitutions.
- MRR and Hits@N are reported on both datasets, with Hits@N measured at N = 1, 5, and 10.MRR is mean reciprocal rank, while Hits@N is the proportion of correct entities ranked in the top N.
- FAAN achieves better performance than traditional KG embedding methods on both NELL and Wiki.
- FAAN consistently outperforms few-shot learning baselines on both datasets across all reported metrics.
- 33.5%/20.6% improvement in MRR/Hits@10 on NELL and 5.6%/10.8% on Wiki are reported against MetaR.
5.6 Impact of Few-Shot Size
FAAN outperforms all baselines across different few-shot sizes on NELL. The results also show that adding more references does not always improve performance, while FAAN remains relatively stable.
- FAAN outperforms all baselines by a large margin under different few-shot sizes K on NELL.
- A larger reference set does not always yield better performance in the few-shot scenario.Performance can be sensitive to which references are available.
- Injecting more location-related references is not necessarily useful for organization-related queries.
- FAAN maintains relatively stable improvements over baselines such as GMatching and FSRL as few-shot size changes.The paper attributes this robustness to better reference embeddings generated by the adaptive aggregator.
5.7 Discussion for Model Variants
Variant experiments examine FAAN's neighbor encoder, Transformer encoder, and matching processor. They support adaptive neighbor aggregation, self-embeddings, Transformer modeling with position embeddings, and adaptive query-reference matching.
- Neighbor Encoder Variants: Adaptive neighbor aggregation and retaining entities' self-embeddings improve model performance.The variants compare mean pooling, fixed attention, and neighbor-only encoding against the adaptive encoder.
- Transformer Encoder Variants: The Transformer encoder effectively models few-shot relations, and position embeddings are essential.
- Matching Processor Variants: The adaptive matching processor has superior capability for computing relevance between references and queries.It is compared with averaging references, selecting the most relevant reference, and an LSTM matching network.
5.8 Case Study for Adaptive Attentions
The case study shows that FAAN's adaptive attentions select task-relevant entity neighbors and query-relevant references. FAAN also performs better than MetaR on most NELL task relations despite substantial relation-level variance.
- Adaptive neighbor attention: FAAN assigns different high-attention neighbors to entities across tasks, with entities focusing more on neighbors related to the task.This supports task-oriented neighbor selection rather than uniform contribution from all neighbors.
- Results across relations: Table 6 compares MetaR and FAAN for each NELL testing relation, with candidate-entity counts and boldface marking the best model results.The table organizes results by relation identifier and reports the number of candidate entities.
- Adaptive reference attention: Reference attention changes with the query: location-related queries emphasize location-related references, while organization-related queries emphasize organization-related references.The adaptive matching processor therefore aggregates references dynamically according to the query.
- Results across relations: FAAN performs better than MetaR on most NELL task relations, indicating robustness across different task relations.Both models show high variance, partly because relations have different candidate-entity set sizes.
6 Conclusion
The paper concludes that FAAN adaptively represents entity pairs and matches references with queries for few-shot KG completion. Experiments on two public datasets show improvements over state-of-the-art methods across different few-shot sizes, while future work includes advanced few-shot relation modeling and textual context.
- Conclusion: FAAN adaptively encodes entity pairs and matches references with queries instead of learning static entity or reference representations.The approach is designed for few-shot KG completion.
- Conclusion: Experiments on two public datasets show that FAAN outperforms current state-of-the-art methods with different few-shot sizes.This is the paper's reported overall experimental conclusion.
- Future work: Future work may use advanced methods for modeling few-shot relations and additional contextual information, such as textual descriptions, to enhance entity embeddings.The authors identify these directions rather than presenting them as part of the current method.