Source-linked AI summary
Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning
Xiangrong Zhu, Guangyao Li, Wei Hu
TL;DR
Federated KG embedding must address data heterogeneity and knowledge forgetting, while existing methods struggle with strongly heterogeneous non-IID data and the authors report no prior exploration of KG embedding unlearning under FL. FedLU uses mutual knowledge distillation for local-global knowledge exchange and combines retroactive interference with passive decay for unlearning. Experiments report better link-prediction accuracy and forgetting without significantly hurting overall performance.
Problem
Federated KG embedding methods struggle with strongly heterogeneous non-IID data, and the authors report no prior exploration of KG embedding unlearning under FL.
Source: §1 · p. 2Method
FedLU uses mutual knowledge distillation to exchange local and global knowledge and combines retroactive interference with passive decay to unlearn specific knowledge and propagate forgetting.
Source: §1 · p. 2, §4.2 · p. 5Results
Takeaways & Limitations
The experiments support FedLU as a framework for federated KG embedding learning and unlearning across three newly constructed heterogeneous datasets.
Source: §6 · p. 8The framework assumes overlapping entities across local KGs but does not assume overlap of relations.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Federated Learning (FL) recently emerges as a paradigm to train a global machine learning model across distributed clients without sharing raw data. Knowledge Graph (KG) embedding represents KGs in a continuous vector space, serving as the backbone of many knowledge-driven applications. As a promising combination, federated KG embedding can fully take advantage of knowledge learned from different clients while preserving the privacy of local data. However, realistic problems such as data heterogeneity and knowledge forgetting still remain to be concerned. In this paper, we propose FedLU, a novel FL framework for heterogeneous KG embedding learning and unlearning. To cope with the drift between local optimization and global convergence caused by data heterogeneity, we propose mutual knowledge distillation to transfer local knowledge to global, and absorb global knowledge back. Moreover, we present an unlearning method based on cognitive neuroscience, which combines retroactive interference and passive decay to erase specific knowledge from local clients and propagate to the global model by reusing knowledge distillation. We construct new datasets for assessing realistic performance of the state-of-the-arts. Extensive experiments show that FedLU achieves superior results in both link prediction and knowledge forgetting.
1 INTRODUCTION
Federated KG embedding combines collaborative learning across distributed knowledge graphs with challenges from heterogeneous client data and the need to forget specific knowledge. FedLU addresses both through mutual knowledge distillation and an unlearning method, and the experiments report improved link prediction and knowledge forgetting.
1 INTRODUCTION
- Client KGs differ in schemas, entity and relation overlap, and graph features, making local embeddings difficult to aggregate and causing drift from global convergence.The introduction links this drift to reduced overall accuracy in federated KG embedding.Source: §1 · p. 2
- Outdated or erroneous KG facts and privacy needs motivate forgetting, but removing a triplet does not remove its effect on trained embeddings, and retraining is costly.The passage also identifies propagation of unlearning to the global model as challenging when local data are not visible globally.Source: §1 · p. 2
- Prior federated KG embedding methods struggle with strongly heterogeneous non-IID data, and the authors report no prior exploration of KG embedding unlearning under FL.The related approaches described include client-server and peer-to-peer architectures.Source: §1 · p. 2
- FedLU uses mutual knowledge distillation to transfer local knowledge to the global model and absorb global knowledge back, and reuses distillation to propagate unlearning.Its unlearning method combines retroactive interference and passive decay to erase specific knowledge from local embeddings.Source: §1 · p. 2
- FedLU outperforms state-of-the-art methods in link prediction and knowledge forgetting in experiments on newly constructed datasets with varied client counts.The introduction identifies three new datasets based on FB15k-237.Source: §1 · p. 2
2 PRELIMINARIES
The preliminaries introduce federated learning and KG embedding, then situate them alongside methods for handling heterogeneity and machine unlearning. They distinguish KG embedding approaches by their scoring or representation strategies.
2 PRELIMINARIES
- FedAvg has clients train local models and a server aggregate them by weighted sum; FedProx adds a proximal term, while MOON adds model-level contrastive loss to address heterogeneity.The passage also notes increasing use of knowledge distillation instead of direct model substitution.Source: §2 · pp. 2–3
- KG embedding places entities and relations in a continuous vector space; approaches include translational-distance, semantic-matching, and deep models.Examples include TransE and RotatE, ComplEx, and convolutional or graph neural networks, respectively.Source: §2 · p. 3
- Machine unlearning seeks to make models forget specific samples; prior approaches use query updates, clustering, or shard-and-slice training, while KG embedding unlearning is reported as unexplored.SISA stores checkpoints while training submodels sequentially on slices of dataset shards.Source: §2 · p. 3
3 RELATED WORK
Related work covers federated KG embedding methods based on client-server and peer-to-peer communication. The approaches differ in what they share and how they address client heterogeneity.
3 RELATED WORK
- FedE adapts FedAvg by sharing entity embeddings, using server delivery and aggregation to combine locally computed embeddings with the global embedding.The described mapping uses client-specific permutation and existence vectors; the passage reports that aggregation can improve performance over locally computed embeddings.Source: §3 · p. 3
- FedR targets clients with high overlap in both entities and relations, sharing relation embeddings globally while representing entity embeddings locally.The paper states that its own setting does not assume relation overlap.Source: §3 · p. 3
- FKGE uses a peer-to-peer differentially private generative-adversarial framework and a translational mapping matrix to align manifolds; this paper avoids peer-to-peer communication for efficiency.Source: §3 · p. 3
- FedEC uses embedding-contrastive learning, adding a contrastive loss to local optimization to increase similarity between local and global embeddings.Source: §3 · p. 3
4 THE PROPOSED FRAMEWORK
FedLU addresses drift in heterogeneous federated KG embedding and supports both embedding learning and forgetting. It pairs mutual knowledge distillation with a two-step unlearning procedure.
4 THE PROPOSED FRAMEWORK
- FedLU is a general federated KG embedding framework whose learning and unlearning modules are designed to maintain performance while forgetting specified knowledge.The framework uses mutual knowledge distillation for learning and combines retroactive interference with passive decay for unlearning.Source: §4 · p. 4
4.1 Learning in FedLU
- To address drift, FedLU keeps local and global embeddings separate and mutually distills their score distributions rather than replacing local models with global ones.Local and global embeddings are refined through mutual distillation, while global aggregation does not force changes to local embeddings.Source: §4.1 · pp. 4–5
4.2 Unlearning in FedLU
- Unlearning first applies hard and soft confusion to forgotten triplets, then uses mutual distillation on retained triples to recover generalization and suppress forgotten-triplet activation.Soft confusion brings the forgotten triplet's score closer to its negative samples, helping prevent overfitting during unlearning.Source: §4.2 · p. 5
5 EXPERIMENTS AND RESULTS
Experiments evaluate FedLU on newly constructed heterogeneous datasets, testing both federated link prediction and unlearning. Further analyses examine knowledge exchange, unlearning components, convergence, and client embeddings.
5.1 Clustering-based Dataset Construction
- Clustering-based partitioning creates more realistic heterogeneous datasets; FB15k-237-C3 has 1,447 overlapping entities versus 11,067 in R3.C3 also has larger and more variable average degrees and clustering coefficients than R3.Source: §5.1 · p. 6
5.2 Experiments on Federated Learning
- FedLU generally achieves better link-prediction performance than competing methods across client counts, and sometimes outperforms centralized training on strongly heterogeneous datasets.FedLU’s global embedding is usually second-best, behind its local embeddings.Source: §5.2 · p. 6, Table 1 · p. 7
5.3 Experiments on Federated Unlearning
- On forgetting sets, FedLU’s unlearned embeddings score lower than raw and retrained embeddings, while retaining test performance comparable to raw models.The evaluation samples 1% of each client’s training triplets as the forgetting set.Source: §5.3 · pp. 6–7, Table 2 · p. 7
5.4 Further Analysis
- Removing mutual knowledge distillation degrades global and local embeddings to FedE and independent-setting performance, respectively.The comparison supports the role of mutual distillation in smooth knowledge exchange.Source: §5.4 · p. 8
- FedLU without hard confusion shows higher Hits@1 on forgetting sets, indicating that hard confusion plays a key role in unlearning.Without soft confusion, Hits@1 is slightly higher than with the original FedLU, suggesting that soft confusion acts as a fine-tuning factor.Source: §5.4 · p. 8
- FedLU converges faster and attains higher average Hits@1 than competitors, which converge more slowly when initializing local training directly from aggregated global embeddings.Figure 4 tracks TransE validation Hits@1 across communication rounds.Source: §5.4 · p. 8, Figure 4 · p. 8
- FedLU’s client embeddings intersect with small overlap and are more clustered within each client, unlike competitors that isolate the smaller client and overlap the larger clients.The authors interpret this pattern as relating and aggregating knowledge across clients while retaining local optimization.Source: §5.4 · p. 8, Figure 5 · p. 8
6 CONCLUSION
FedLU combines mutual knowledge distillation for federated KG embedding learning with an unlearning method that combines retroactive interference and passive decay.
6 CONCLUSION
- Experiments on three newly constructed heterogeneous datasets show better link-prediction accuracy and forgetting of specific knowledge without significantly hurting overall performance.Future work will study continual learning and unlearning across the KG life cycle.Source: §6 · p. 8
A DATASET CONSTRUCTION ALGORITHM
The algorithm constructs client datasets by clustering relations and assigning each triplet according to its relation cluster.
A DATASET CONSTRUCTION ALGORITHM
- The algorithm maps entities to linked relations, builds a relation co-occurrence matrix, spectrally clusters relations, then assigns triplets to clients by cluster.It uses the first |K| eigenvectors of the Laplacian matrix followed by k-means to form |K| relation clusters.Source: A DATASET CONSTRUCTION ALGORITHM · p. 10
B DATASET COMPARISON
The appendix compares degree distributions for the clustering-based and random-partition datasets with five and ten clients.
B DATASET COMPARISON
- Degree distributions vary more across clients in FB15k-237-C5/C10 than in FB15k-237-R5/R10, similar to the C3-versus-R3 comparison.Source: B DATASET COMPARISON · p. 10
C.1 Learning Results on FB15k-237-R3/R5/R10
On FB15k-237-R3/R5/R10, FedLU consistently outperforms the other competitors and sometimes beats centralized learning. The results also show that the relationship between local and global embeddings varies with dataset heterogeneity.
C.1 Learning Results on FB15k-237-R3/R5/R10
- FedLU outperforms other competitors stably on all three datasets and sometimes beats the centralized setting.Both FedLU and existing federated KG embedding methods generally improve over the independent setting.Source: §C.1 · p. 10
- On the less heterogeneous R3/R5/R10 datasets, the gap between independent and centralized settings is more significant.The paper attributes this to the datasets being easier to improve with more data.Source: §C.1 · p. 10
- FedLU (local) generally underperforms FedLU (global) on R3/R5/R10, unlike on Table 1, because these datasets are nearly homogeneous.The paper says global embeddings represent the entire data and may not capture local heterogeneous data well.Source: §C.1 · p. 10
C.2 Unlearning Results on FB15k-237-C5/C10
On FB15k-237-C5/C10, FedLU rapidly lowers results on forgetting sets while causing only slight declines on test sets. Its client embeddings are also better clustered than competitors’ in the FB15k-237-C5 visualization.
C.2 Unlearning Results on FB15k-237-C5/C10
- On FB15k-237-C5/C10, FedLU rapidly lowers results on forgetting sets while causing only slight declines on test sets.The reported averages are shown for C5 and C10 in Tables 5 and 6.Source: §C.2 · p. 10, Table 5 · p. 11, Table 6 · p. 11
- Retrained results on forgetting sets remain much higher than FedLU’s unlearning results, indicating that retraining on retaining sets alone does not achieve unlearning.The paper presents this comparison as validation of its unlearning method’s effectiveness.Source: §C.2 · p. 10, Table 5 · p. 11, Table 6 · p. 11
C.3 Entity Embedding Visualization on FB15k-237-C5
- In the FB15k-237-C5 TransE visualization, FedLU’s entity embeddings on each client are better clustered than those of other competitors.Figure 7 maps the client embeddings using t-SNE.Source: §C.3 · p. 10