PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Subgraph Federated Learning

Jinheon Baek, Wonyong Jeong, Jiongdao Jin, Jaehong Yoon, Sung Ju Hwang

arXiv:2206.10206v3cs.LG

TL;DR

Private subgraphs may have missing links, and heterogeneity across communities can make conventional subgraph FL aggregation collapse incompatible local knowledge. FED-PUB estimates model similarities with functional embeddings on random graphs and uses personalized sparse masks for selective parameter sharing. It significantly outperforms relevant baselines across six datasets covering overlapping and non-overlapping subgraphs.

  • Problem

    Subgraph FL must address missing links between private subgraphs while accounting for community-level heterogeneity that can make naive aggregation collapse incompatible knowledge.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    FED-PUB uses functional embeddings of local GNNs on random graphs to weight server aggregation, then applies a personalized sparse mask to select locally relevant parameters.

    Source: §1 · p. 2, §3 · p. 4, §4.2 · p. 5

  • Results

    FED-PUB significantly outperforms relevant baselines across six datasets in overlapping and non-overlapping subgraph scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Personalized subgraph FL targets joint improvement of interrelated local models rather than learning a single global model.

    Source: §1 · p. 2

    Direct parameter-vector similarity is limited by high-dimensional distance becoming uninformative and similarity-computation cost growing with model size.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Subgraphs of a larger global graph may be distributed across multiple devices, and only locally accessible due to privacy restrictions, although there may be links between subgraphs. Recently proposed subgraph Federated Learning (FL) methods deal with those missing links across local subgraphs while distributively training Graph Neural Networks (GNNs) on them. However, they have overlooked the inevitable heterogeneity between subgraphs comprising different communities of a global graph, consequently collapsing the incompatible knowledge from local GNN models. To this end, we introduce a new subgraph FL problem, personalized subgraph FL, which focuses on the joint improvement of the interrelated local GNNs rather than learning a single global model, and propose a novel framework, FEDerated Personalized sUBgraph learning (FED-PUB), to tackle it. Since the server cannot access the subgraph in each client, FED-PUB utilizes functional embeddings of the local GNNs using random graphs as inputs to compute similarities between them, and use the similarities to perform weighted averaging for server-side aggregation. Further, it learns a personalized sparse mask at each client to select and update only the subgraph-relevant subset of the aggregated parameters. We validate our FED-PUB for its subgraph FL performance on six datasets, considering both non-overlapping and overlapping subgraphs, on which it significantly outperforms relevant baselines. Our code is available at https://github.com/JinheonBaek/FED-PUB.

1. Introduction

Private subgraphs may lack cross-subgraph edges, but differences between graph communities also make indiscriminate model aggregation problematic. FED-PUB instead shares knowledge selectively among interrelated local models and significantly outperforms relevant baselines across six datasets.

1. Introduction

  • Subgraphs from different communities can have opposite properties, so aggregating their local models may collapse incompatible knowledge.Existing subgraph FL methods address missing edges, but may overlook this heterogeneity.Source: §1 · p. 1
  • FED-PUB estimates local-model similarities with functional embeddings on random graphs, then uses weighted server aggregation and personalized sparse masks to select relevant parameters.The server cannot access each client’s subgraph, and similarity alone does not identify which aggregated parameters are relevant locally.Source: §1 · p. 2
  • Personalized subgraph FL aims to jointly improve interrelated local models, such as models for subgraphs in the same community, through selective knowledge sharing.The goal differs from learning a single global model.Source: §1 · p. 2
  • FED-PUB significantly outperforms relevant baselines across six datasets and overlapping and non-overlapping subgraph scenarios.The evaluation varies the number of clients and includes real-world datasets.Source: §1 · p. 2

2. Related Work

Related work covers federated learning for distributed data and graph-level or subgraph-level GNN training. Subgraph FL must contend with missing links and community structure, while prior edge-augmentation approaches may raise privacy and communication concerns.

2. Related Work

  • Federated learning aggregates locally trained client models, while personalization methods such as FedPer retain local layers when client data are heterogeneous.FedProx instead regularizes differences between local and global models.Source: §2 · p. 2
  • Subgraph FL addresses probable missing links between local portions of a larger graph, a structural challenge absent from methods assuming clients have disjoint graphs.Prior approaches augment subgraphs with nodes from other clients, which may compromise privacy and increase communication overhead.Source: §2 · p. 3
  • Graph-level FL studies disjoint graphs and non-IID graph labels, whereas subgraph-level FL concerns missing links between parts of a larger graph.Graph-structured data also presents community structures between private subgraphs.Source: §2 · p. 3
  • Existing subgraph FL methods augment nodes using information from other subgraphs, but this may compromise privacy and increase communication overhead.FED-PUB instead explores communities of densely connected subgraphs.Source: §2 · p. 3

3. Problem Statement

Subgraph FL must address missing cross-client edges and community-level heterogeneity; this paper formulates personalization as collaboration among related local models rather than one universal model.

3. Problem Statement

  • GNNs represent nodes using their own and neighboring-node features, while federated learning trains collaboratively without exposing clients' local data.FedAvg aggregates local knowledge according to each client's number of training instances.Source: §3 · p. 3
  • Subgraph FL must address missing edges between clients' local graphs, while existing node-augmentation approaches can incur high communication costs and privacy concerns.GNNs use neighboring-node relationships, and some methods estimate nodes from other clients' subgraphs to address missing links.Source: §3 · p. 3
  • Subgraphs in different communities can be heterogeneous, so a universal parameter set may be suboptimal and naive FL may collapse incompatible knowledge.Subgraphs within a community tend to have similar properties, while subgraphs in opposite communities may not.Source: §3 · p. 4
  • Personalized subgraph FL aims to jointly improve related local models by sharing weights among subgraphs in the same community.The formulation targets interrelated subgraphs rather than a single parameter set intended to work across all subgraphs.Source: §3 · p. 4
  • The formulation weights aggregation between clients with α_ij and applies a trainable sparse mask μ_i to filter weights irrelevant to client i's subgraph.The mask is intended to shift and filter parameters across heterogeneous communities.Source: §3 · p. 4

4. Federated Personalized Subgraph Learning

FED-PUB estimates similarities between local GNNs without accessing client subgraphs, then uses similarity-weighted aggregation and client-local masks to personalize shared parameters. A proximal term further regularizes locally updated models toward the server model.

4.1. Subgraph Similarity Estimation

  • Random-graph functional embeddings estimate GNN similarity without client data access, enabling community-aware parameter sharing between related subgraphs.The method compares outputs for the same random-graph input; parameter-vector similarity is described as unreliable in high dimensions and increasingly costly with model size.Source: §4.1 · pp. 4–5, §4 · p. 4
  • Weighted averaging uses functional similarities across all clients, avoiding both indiscriminate global aggregation and exclusive reliance on local objectives.The similarity scaling hyperparameter τ controls the weighting; larger values make averaging nearly exclusive to subgraphs identified as belonging to the same community.Source: §4.1 · p. 5
  • Sharing knowledge among likely linked subgraphs within a community can implicitly address missing edges while enhancing privacy and minimizing communication costs.Source: §4.1 · p. 5

4.2. Adaptive Weight Masking

  • Each client applies an unshared sparse local mask to aggregated GNN weights, selecting parameters for local training and potentially reducing communication and runtime.Masks start as ones and are encouraged to be sparse with an L1 regularizer; faster local execution depends on support for zero-skipping operations.Source: §4.2 · p. 5
  • A proximal term regularizes each locally updated model toward the server model to limit drift toward the client’s local data distribution.Source: §4.2 · pp. 5–6

5. Experiments

Experiments on six graph datasets examine FED-PUB across overlapping and non-overlapping subgraphs, primarily for node classification and also for link prediction. FED-PUB outperforms baselines across the reported settings, with analyses of convergence, community detection, masking, and communication costs.

5.1. Experimental Setups

  • The evaluation uses six datasets, METIS partitions, and overlapping or non-overlapping subgraphs, comparing FED-PUB with federated and local baselines.The experiments focus mainly on node classification and also include link prediction.Source: §5 · p. 6, §5.1 · p. 6

5.2. Experimental Results

  • FED-PUB outperforms all baselines in both overlapping and non-overlapping node settings.Source: Table 2 · p. 7, §5.2 · p. 7
  • FED-PUB converges rapidly against baselines, while five local epochs perform worse than one, showing that more local training is not always beneficial.With more communication rounds and local steps, models can overfit their local subgraphs; the proximal term alleviates this issue.Source: §5.2 · pp. 7–8, Figure 8 · p. 8
  • Functional similarities identify four clear communities early and later capture a larger community composed of two smaller ones.The heatmaps compare missing-edge patterns, label similarities, and FED-PUB’s functional similarities.Source: §5.2 · p. 8, Figure 6 · p. 8
  • Similarity matching and weight masking each improve over FedAvg, with their relative benefits depending on whether subgraphs overlap.Similarity matching helps more in the overlapping setting, while personalized masks help more in the non-overlapping setting.Source: §5.2 · p. 8, Figure 7 · p. 8
  • Sparse masks reduce communication costs and model sizes; the sparsity-performance trade-off can be adjusted with λ1.Existing subgraph FL methods require more than twice FedAvg’s communication costs, according to the reported comparison.Source: §5.2 · p. 8, Table 3 · p. 8
  • On neighboring subgraphs with the most missing edges, FED-PUB achieves superior performance to subgraph FL baselines without explicitly augmenting them.The neighbor measure evaluates a client’s subgraph using its neighbor model; strong performance indicates shared knowledge across subgraphs.Source: §5.2 · pp. 8–9, Figure 9 · p. 8
  • FED-PUB also outperforms baselines on imbalanced subgraphs and link prediction, with rapid link-prediction convergence for 5- and 10-client arxiv settings.Source: §5.2 · p. 9, Figure 10 · p. 9

6. Conclusion

FED-PUB jointly improves local GNNs on interrelated subgraphs through similarity-based aggregation and client-specific masking. The appendices describe its algorithms and experimental setups.

6. Conclusion

  • FED-PUB estimates subgraph similarities with functional embeddings, aggregates local models by similarity, and masks weights to focus on each community’s relevant subnetwork.The framework targets joint improvement of local GNNs on interrelated subgraphs rather than a single global model.Source: §6 · p. 9

A. Algorithms

  • The client algorithm applies its mask to received weights before local training, while the server algorithm performs similarity matching.The appendix identifies client-side weight masking and server-side similarity matching as the two algorithm components.Source: A. Algorithms · p. 13

B. Experimental Setups

  • The experimental-setup appendix covers six benchmark datasets, baselines and FED-PUB, plus implementation details for synthetic and real-world graph experiments.It also describes additional experiments on functional similarities and sparse masks.Source: B. Experimental Setups · p. 13

B.1. Datasets

The experiments use six benchmark datasets with overlapping and non-overlapping subgraph settings, and compare FED-PUB with federated, personalized, graph, and local baselines.

B.1. Datasets

  • The six datasets span citation graphs and Amazon product graphs, with statistics covering subgraph size, classes, clustering, and label-distribution heterogeneity.The datasets are Cora, CiteSeer, Pubmed, ogbn-arxiv, Computer, and Photo.Source: §B.1 · p. 14
  • Non-overlapping clients receive METIS partitions, while overlapping clients receive sampled half-subgraphs from METIS partitions, repeated five times per partition.The overlapping setup uses 2, 6, or 10 initial partitions for 10, 30, or 50 clients, respectively.Source: §B.1 · p. 14

B.2. Baselines and Our Model

  • The comparison includes FedAvg and FedProx, personalized FedPer, subgraph methods FedGNN and FedSage+, graph-FL method GCFL, and local training.These baselines cover standard aggregation, personalization, subgraph expansion, client grouping, and no weight sharing.Source: §B.2 · p. 14
  • FED-PUB estimates subgraph similarities from model functional embeddings to discover communities and adaptively masks server weights to filter weights irrelevant to heterogeneous communities.The listed model is evaluated alongside the baselines in both overlapping and non-overlapping scenarios.Source: §B.2 · p. 14

B.3. Implementation Details

The appendix specifies how FED-PUB generates functional embeddings and sparsifies client masks, reports common experimental settings, and examines hyperparameter trade-offs.

B.3. Implementation Details

  • Functional embeddings use five random 100-node subgraphs sampled from a stochastic block model, with within-graph edge probability 0.1 and between-graph probability 0.01.Node features are initialized from a normal distribution with mean 0.0 and variance 1.0; the server distributes the sampled graph to clients.Source: §B.3 · p. 15
  • Training encourages sparse client masks with an L1 penalty, and test-time thresholding sets mask elements below λ2 to zero, reducing communication and prediction costs.The passage says thresholding permits transmitting and predicting with only partial parameters.Source: §B.3 · p. 15
  • Experiments use two-layer GCNs with a 128-dimensional hidden layer, learning rate 0.001, Adam, and participation by all clients each round.The default λ1 and λ2 are both 0.001; real-world runs use different local epochs and rounds by dataset size and node-overlap setting.Source: §B.3 · p. 15

C.1. Results on Varying Scaling Hyperparameters in Loss Function

  • Higher λ1 increases model sparsity while slightly decreasing performance, whereas sufficiently large λ2 values improve performance without changing sparsity.The analysis uses Cora with overlapping nodes, 10 clients, and 2 local epochs; λ2 is described as limiting excessive drift toward the local subgraph distribution.Source: §C.1 · p. 16, Table 5 · p. 16

C.2. Results on Louvain Graph Partitioning Algorithm

Under Louvain partitioning, FED-PUB outperforms all reported baselines on three datasets. The authors note that Louvain cannot specify the number of partitions, limiting how partitions are assembled for clients.

C.2. Results on Louvain Graph Partitioning Algorithm

  • Louvain produces 38 CiteSeer partitions, including three with fewer than ten nodes, and cannot specify the partition count for client assignment.The setup randomly merges partitions to create 10 clients without considering graph properties, potentially mixing structurally distinct subgraphs.Source: §C.2 · p. 16
  • The authors consider METIS more appropriate for these experiments because it can specify the number of partitions, unlike Louvain.They note that random merging of Louvain partitions can combine subgraphs with distinct structural roles.Source: §C.2 · p. 16
  • FED-PUB consistently outperforms all other baselines on Cora, CiteSeer, and PubMed with 10 clients under Louvain partitioning.The results are reported in Table 6.Source: §C.2 · p. 16, Table 6 · p. 16

C.3. Results on Random Graph Partitioning Algorithm

These analyses test FED-PUB under random partitions and examine how subgraph heterogeneity relates to its masking and aggregation strategies. FED-PUB remains ahead of baselines across the reported settings, while performance gaps vary with community structure and client count.

C.3. Results on Random Graph Partitioning Algorithm

  • FED-PUB consistently outperforms all baselines on random CiteSeer partitions, although the gap narrows relative to overlapping and non-overlapping community-based splits.The random partitions lack specific community structures; the example partition has 211 nodes and 72 edges per subgraph.Source: §C.3 · p. 17, Table 7 · p. 17

C.4. Analyses on Distribution Shifts Between Subgraphs with Sparse Masks

  • Label-distribution divergence within communities is 0.384 for non-overlapping and 0.047 for overlapping subgraphs, indicating greater within-community heterogeneity in the non-overlapping setup.Between-community divergence is 0.639 for non-overlapping and 0.528 for overlapping subgraphs.Source: §C.4 · p. 17
  • Masking brings large gains in the high-heterogeneity non-overlapping setting, while personalized aggregation is more beneficial in the lower-heterogeneity overlapping setting.With three clients, FED-PUB still outperforms all baselines by large margins, though its performance gaps are reduced versus settings with 5, 10, or 20 clients.Source: §C.4 · p. 17, Table 8 · p. 17, §C.5 · p. 18
  • For the most similar Cora subgraphs, 75% of parameters are shared; for the most heterogeneous pair, masks filter 73%, leaving 27% shared.The analysis uses 20 clients in the non-overlapping setting.Source: §C.4 · p. 17

C.5. Analyses on Local Graph Size vs Heterogeneity

  • Subgraph heterogeneity becomes more severe as client count increases in both overlapping and non-overlapping scenarios.The analysis links this issue to FED-PUB’s sparse local masks.Source: §C.5 · p. 17

C.6. Results on Varying Graph Inputs for Functional Embeddings

FED-PUB estimates functional similarities using shared random graph inputs, on the assumption that random structures avoid bias from any particular client subgraph. Experiments compare random graph generators and other input choices for those embeddings.

C.6. Results on Varying Graph Inputs for Functional Embeddings

  • FED-PUB uses the same random graph inputs for every client, assuming randomness avoids bias from any particular subgraph’s graph structure.The graphs use stochastic block model initialization and normally distributed node features.Source: §C.6 · p. 18
  • On Cora, SBM and ER inputs capture subgraph similarities more accurately than a one-node input, indicating that sufficient randomness is needed to capture the model’s functional space.The comparison covers overlapping and non-overlapping scenarios with 20 clients.Source: §C.6 · p. 18, Table 9 · p. 18
  • SBM and ER random inputs outperform existing client node features for capturing subgraph similarities, supporting the use of random inputs to avoid bias.Source: §C.6 · p. 18, Table 9 · p. 18

C.7. Results on Varying Similarity Estimation Mechanisms

The analyses compare similarity-estimation and community-aggregation choices in FED-PUB, and examine performance under missing edges. Functional similarities perform comparably to label similarities, implicit community modeling generally wins, and the Oracle remains strongest.

C.7. Results on Varying Similarity Estimation Mechanisms

  • On overlapping Cora, functional and label similarity schemes outperform parameter- and gradient-based schemes, while functional similarity performs similarly to label similarity.Label distributions are privacy-sensitive local information, whereas functional embeddings use privacy-free random graphs.Source: §C.7 · p. 18, Table 10 · p. 18

C.8. Analyses on Implicit and Explicit Communities in Weight Aggregation

  • Implicit community modeling during weight aggregation outperforms explicit grouping in all but one reported case, Cora with overlapping nodes.The authors suggest that ignoring other communities may help in that exceptional case.Source: §C.8 · p. 19, Table 11 · p. 19

C.9. Impacts of Missing Edges to Performance Degeneration

  • The Oracle outperforms all methods, while FED-PUB achieves the closest performance to it in the reported missing-edge comparison.The Oracle trains on the global graph including missing edges, unlike FL methods trained with only within-subgraph edges.Source: §C.9 · p. 19, Table 12 · p. 19

D. Discussion on Limitations and Potential Societal Impacts

The discussion notes that experiments mainly use unipartite graphs and that transmitted model parameters may contain privacy-sensitive information. It also describes potential applications of FED-PUB in graph-based domains where interaction data can remain local.

Limitations

Potential Societal Impacts

  • Potential applications include social, recommendation, and patient networks, with interaction data kept locally while clients share trained models.The authors expect this approach could alleviate risks associated with storing user interaction data on a server.Source: D. Discussion on Limitations and… · p. 20
  • Transmitted model parameters may contain privacy-sensitive information, an issue the paper assumes away rather than addressing as a main focus.The authors suggest further study of parameter safety and differential privacy techniques if needed.Source: D. Discussion on Limitations and… · p. 20

Related papers

Loading 2206.10206v3…