PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Graph Neural Network Training: A Survey

Yingxia Shao, Hongzheng Li, Xizhi Gu, Hongbo Yin, Yawen Li, Xupeng Miao, Wentao Zhang, Bin Cui, Lei Chen

arXiv:2211.00216v2cs.LGcs.AIcs.DBcs.DC

TL;DR

Large graphs strain single-device GNN training, while graph dependencies complicate distributed execution. This survey organizes optimization techniques into a four-category taxonomy and reviews representative systems. Its synthesis identifies accuracy, communication, and workload balance as central challenges.

  • Problem

    Large graphs exceed the memory and computing resources of a single device, while graph dependencies complicate efficient distributed GNN training.

    Source: §1 · p. 2, §3.2 · p. 7

  • Method

    The survey reviews distributed GNN techniques using a lifecycle-based taxonomy of data partition, batch generation, execution model, and communication protocol.

    Source: §1 · p. 3

  • Results

    The survey organizes optimization techniques into four categories and reviews 28 representative distributed GNN systems and frameworks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The taxonomy maps techniques across the end-to-end training lifecycle and provides a structured overview of distributed GNN systems.

    Source: §1 · p. 3

    Broadcast efficiency can be undermined by workload imbalance, which causes computation stalls.

    Source: §7.1.1 · p. 25

Abstract

from arXiv · show

Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs, since it is able to provide abundant computing resources. However, the dependency of graph structure increases the difficulty of achieving high-efficiency distributed GNN training, which suffers from the massive communication and workload imbalance. In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed. Yet, there is a lack of systematic review on the optimization techniques for the distributed execution of GNN training. In this survey, we analyze three major challenges in distributed GNN training that are massive feature communication, the loss of model accuracy and workload imbalance. Then we introduce a new taxonomy for the optimization techniques in distributed GNN training that address the above challenges. The new taxonomy classifies existing techniques into four categories that are GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. We carefully discuss the techniques in each category. In the end, we summarize existing distributed GNN systems for multi-GPUs, GPU-clusters and CPU-clusters, respectively, and give a discussion about the future direction on distributed GNN training.

1 INTRODUCTION

GNNs are useful across many domains, but large graphs exceed the efficient reach of many single-machine approaches. This section introduces distributed GNN training and a lifecycle-based taxonomy for its optimization techniques.

1 INTRODUCTION

  • GNNs have applications across social networks, bioinformatics, drug discovery, traffic forecasting, health care, and recommendation.The survey notes that GNN models integrate graph structure into deep learning, while many existing frameworks and optimizations focus on single-machine training.Source: §1 · p. 2
  • Large graphs can contain billions of vertices and trillions of edges, making existing GNN models inefficient or impossible to run on them.A single device's memory and computing resources become bottlenecks, motivating distributed training across resources such as multiple GPUs or CPU clusters.Source: §1 · p. 2
  • Earlier surveys addressed GNN models, parallel computing, hardware accelerators, or system evolution, while systematic review of distributed-training techniques remained limited.The authors distinguish their focus from Besta et al. [10], Abadal [1], and Vatter et al. [111].Source: §1 · p. 3
  • The survey organizes distributed GNN training around data partition, batch generation, and model training, then classifies optimizations into four categories.The taxonomy covers data partition, batch generation, execution model, and communication protocol across mini-batch and full-graph training.Source: §1 · p. 3
  • The survey reviews distributed GNN training techniques and summarizes 28 representative systems and frameworks from industry and academia.It also discusses future directions for distributed GNN training.Source: §1 · p. 3

2 PRELIMINARIES OF GRAPH NEURAL NETWORKS

GNN layers update vertex representations using neighborhood information, creating dependencies that distinguish their training from conventional neural networks. These dependencies affect mini-batch construction and distributed execution.

Graph and graph partition

  • A graph consists of vertices and edges; edge-cut partitions divide vertices, while vertex-cut partitions divide edges and may replicate vertices.Replicated vertices in a vertex-cut partition have replication factor greater than one and are boundary vertices.Source: §2 · pp. 3–4

Graph neural networks

  • Each GNN layer updates vertex features by aggregating neighborhood information and applying model transformations.The global formulation uses hidden embeddings, normalized adjacency, model weights, and a nonlinear function; the local formulation defines aggregation and update functions and may include edge embeddings.Source: §2 · p. 4
  • Graph data dependency arises because each vertex's hidden embedding depends on its neighbors' previous-layer embeddings, enabling structural information to be captured.The paper relates this dependency to tasks such as node classification and link prediction.Source: §2 · p. 4

GNN training methods

  • Mini-batch GNN training updates parameters multiple times per epoch, and exact batches may require each training vertex's L-hop neighbors.These batches can grow to include a substantial portion of the graph, so methods sample neighborhoods to reduce their size.Source: §2 · pp. 4–5
  • GNN computation adds neighborhood aggregation in forward passes and gradient scatter along edges in backward passes.These operations make computation more complex than in DNN training.Source: §2 · p. 5

Distributed DNN training

  • Distributed DNN training commonly manages replicated or partitioned models and synchronizes gradients, while GNN training must also handle graph-data dependencies.GNN models are typically shallow, but their potentially enormous input graphs make communication overhead a major concern.Source: §2 · pp. 5–6

3 DISTRIBUTED GNN TRAINING AND CHALLENGES

Distributed GNN training divides work among workers, but graph dependencies complicate partitioning, batch generation, and model execution. This section frames the resulting challenges and organizes proposed techniques by training-pipeline stage.

3.1 General Distributed GNN Training Pipeline

  • The pipeline partitions graph and feature data, optionally generates mini-batches, executes the GNN, and updates parameters; execution and communication requirements differ between mini-batch and full-graph training.Cross-worker edges create dependencies: respecting them increases communication, while ignoring them can damage accuracy.Source: §3.1 · pp. 6–7, Figure 2 · p. 6

3.2 Challenges in Distributed GNN Training

  • Distributed GNN training faces massive feature communication, accuracy loss from approximate mini-batches, and workload imbalance that leaves workers waiting.Mini-batch approximations can improve efficiency but do not guarantee convergence; full-graph training is described as convergence-guaranteed and able to match single-worker accuracy.Source: §3.2 · p. 7

3.3 Taxonomy of Distributed GNN Training Techniques

  • The survey groups optimization techniques by pipeline stage into data partition, batch generation, execution model, and communication protocol.The taxonomy is intended to organize techniques that optimize the same stage and clarify solutions across distributed GNN training.Source: §3.3 · p. 8, Figure 4 · p. 9
  • Data-partition techniques seek to balance workload and reduce communication, using GNN-oriented cost models and sometimes partitioning graph structure and features independently.Traditional graph partitioning methods can be used, but the survey says they are not optimal for GNN workloads.Source: §3.3 · p. 8
  • Distributed mini-batch generation methods include distributed sampling and partition-based generation, with caching also used to reduce communication.Standard sampling can cause frequent cross-worker data access because it does not account for distributed settings.Source: §3.3 · pp. 8–9
  • Execution-model techniques schedule mini-batch sampling and feature extraction or organize full-graph aggregation, while communication protocols govern access to remote embeddings.Full-graph execution is categorized by matrix or graph views, operator order, and synchronization timing; protocols may be synchronous or asynchronous, with asynchronous methods using stale embeddings.Source: §3.3 · pp. 9–10

4 GNN DATA PARTITION

The survey organizes data partition around estimating GNN workload costs, partitioning graphs, and partitioning features to balance workload and reduce communication. Its techniques account for GNN-specific factors that traditional graph partitioning may miss.

4 GNN DATA PARTITION

  • Data partition methods cover graph and feature data, with workload balance and communication and computation minimization as objectives.A cost model estimates partition workloads to guide graph partitioning.Source: §4 · p. 10, Figure 5 · p. 10

4.1 Cost model of GNN

  • GNN-specific cost models estimate computation and communication using factors beyond vertex and cross-edge counts, including feature dimensions, layer count, and training-vertex distribution.The survey groups these models into heuristic, learning-based, and operator-based approaches.Source: §4.1 · p. 11

4.2 Graph partition in GNN

  • For mini-batch training, graph partition methods balance training vertices across workers and may also preserve L-hop neighborhoods or minimize computation cost.Approaches include multi-constraint and streaming partitioning; one computation-cost function is submodular and supports a 2 − 2/M approximation.Source: §4.2 · p. 13

4.3 Feature partition in GNN

  • Feature partitioning can be independent of graph partitioning; column-wise partitioning distributes feature dimensions across partitions, especially for high-dimensional inputs and low-dimensional hidden features.Other approaches align feature partitioning with 2D graph partitioning or use row-wise partitioning with replication.Source: §4.3 · p. 14

5 GNN BATCH GENERATION

Distributed batch generation addresses communication and accuracy challenges through distributed sampling or by using local partitions as mini-batches. The survey reviews caching and sampling optimizations as well as methods that expand partitions or apply global correction.

5 GNN BATCH GENERATION

  • Distributed mini-batch generation either samples across workers or uses each local partition as a batch to address communication and accuracy challenges.The survey frames these as two approaches to distributed batch generation.Source: Figure 6 · p. 14, §5 · p. 15

5.1 Mini-batch generation with distributed sampling

  • Distributed sampling can trigger frequent remote data access, while cache strategies store frequently accessed remote vertices locally to reduce communication.The survey describes static, dynamic, and sampling-informed cache policies, alongside techniques that prioritize local sampling or send sampling tasks to remote workers.Source: §5.1 · pp. 15–16

5.2 Partition-based mini-batch generation

  • Partition-based mini-batches avoid massive communication by restricting each worker to its local partition, but ignoring cross-partition edges can reduce model accuracy.Subgraph expansion replicates remote boundary neighbors locally; LLCG periodically applies global correction, which empirical results show can significantly improve accuracy.Source: §5.2 · pp. 16–17

6 GNN EXECUTION MODEL

The survey organizes execution models around distinct scheduling challenges in mini-batch and full-graph training. It compares graph-operator and matrix views, then distinguishes synchronous execution from asynchronous approaches using historical states or parallel mini-batches.

6.1 Mini-batch Execution Model

  • Mini-batch sampling and feature extraction consume almost 83%–99% of overall training time, with communication and resource contention identified as causes of low efficiency.Conventional execution performs sampling and feature extraction for each batch before training it; the surveyed alternatives target this bottleneck.Source: §6.1 · p. 17
  • Operator-parallel execution pipelines batch-generation operators across mini-batches, while factored execution assigns operations to dedicated devices or isolated resources to reduce contention.Operator-level scheduling can balance CPU sampling and GPU feature extraction; factored designs can allocate sampling GPUs separately from feature-extraction and training GPUs.Source: §6.1 · pp. 17–18
  • Pull-push parallelism replaces costly input-feature movement with lighter graph-structure movement, then uses model-parallel aggregation followed by data-parallel mini-batch training.Workers pull remote vertices and edges, construct subgraphs locally, and exchange low-dimensional hidden embeddings between stages.Source: §6.1 · pp. 18–19

6.2 Full-graph execution model

  • Full-graph execution is difficult to parallelize efficiently because a computation graph is partitioned among workers whose computations depend on one another.The survey treats vertex computation and update mode as intersecting perspectives, so a system adopts one model from each.Source: §6.2 · p. 19

6.2.1 Graph View.

  • Graph-view execution divides neighborhood aggregation into one-shot collection or chunk-based partial aggregations, processed sequentially or in parallel.One-shot execution collects all needed neighbor features in advance; chunking addresses its memory and communication costs, with sequential processing addressing out-of-memory risk and parallel processing reducing communication volume.Source: §6.2.1 · p. 20

6.2.2 Matrix View.

  • Distributed SpMM is classified as computation-only, communication-computation, or communication-computation-reduction according to matrix partitioning and which matrix remains stationary.The partition and stationary strategies determine whether communication and reduction stages are needed; computation-only execution requires replicating one of A or H, limiting scalability when both exceed worker memory.Source: §6.2.2 · p. 21, Table 2 · p. 22

6.2.3 Comparison of graph view and matrix view.

  • The graph view is a higher-level programming interface, while the matrix view directly manages matrices; one-shot maps to 1D P-Stationary SpMM and chunked partial results map to reductions.Graph operations such as feature aggregation are ultimately transformed into matrix multiplications.Source: §6.2.3 · p. 23

6.2.4 Synchronous execution model.

  • Synchronous execution blocks at GA and ▽GA until boundary-vertex communication finishes, whereas asynchronous execution can use historical embeddings or gradients or pipeline mini-batches.Type I removes aggregation-stage waits using historical states; type II removes weight-update synchronization, and the two can be used individually or jointly.Source: §6.2.4 · p. 23, §6.2.5 · pp. 23–24

7 GNN COMMUNICATION PROTOCOLS

Synchronous communication protocols transfer the embeddings and gradients needed by distributed execution. The survey contrasts broad broadcast, targeted point-to-point exchange, pipelined aggregation, and retrieval through shared memory.

7.1.1 Broadcast-based protocol.

  • Broadcast shares each worker’s latest hidden embeddings or graph structure across workers, but workload imbalance can stall computation and reduce broadcast efficiency.CAGNET applies broadcast to partial graph-structure or feature matrices under several graph partitioning strategies.Source: §7.1.1 · pp. 24–25

7.1.2 Point-to-point-based protocol.

  • Point-to-point transmission sends only data associated with cross-partition edges or replicated vertices, reducing communication volume relative to broadcasting to all workers.Naive point-to-point exchange can add request-and-send latency; pipeline-based protocols are introduced to overlap communication and local computation.Source: §7.1.2 · p. 25

7.1.3 Pipeline-based protocol.

  • Pipeline-based protocols divide vertex aggregation into subtasks and overlap their communication and computation.Chunk aggregation can serve as a subtask; remote partial aggregation can reduce transferred data, while FlexGraph overlaps remote transfers with local partial aggregation.Source: §7.1.3 · pp. 25–26

7.1.4 Communication via Shared Memory.

  • Shared-memory approaches store graph data and embeddings in CPU memory and use GPU device memory as a cache, with distributed shared memory serving larger datasets.NeuGraph fetches edge and vertex chunks from CPU memory and uses topology-aware chain scheduling to avoid bandwidth contention during transfers.Source: §7.1.4 · p. 26

7.2 Asynchronous GNN communication protocol

Asynchronous protocols avoid synchronization by using historical information, so the survey organizes them around bounded staleness and the mechanisms that realize it. The approaches vary in whether they use old embeddings, gradients, or both, and whether computation proceeds vertex by vertex without stale values.

7.2.1 Staleness Models for Asynchronous GNN communication protocol.

  • Asynchronous communication uses historical information, and staleness models bound its age or variation to support convergence.The survey distinguishes epoch-fixed, epoch-adaptive, and variation-based bounds.Source: §7.2.1 · pp. 26–27
  • Epoch-fixed staleness bounds the historical information’s epoch gap by the user-set hyperparameter ε_E.Epoch-adaptive staleness permits different gaps across neighbors and epochs, subject to ε_A; variation-based staleness bounds embedding or gradient change by ε_V.Source: §7.2.1 · p. 27

7.2.2 Realization of Asynchronous GNN communication protocols.

  • Realizations range from skipped broadcasts or periodic embedding pushes to asynchronously transmitted partial aggregations with a bounded iteration delay.Some protocols use historical embeddings only at GA, while others also use stale gradients at ▽GA.Source: §7.2.2 · pp. 27–28, §7.2.1 · p. 27
  • Some pipelines use both stale embeddings and gradients, with user-bounded staleness; vertex-level asynchronization instead starts each vertex when its neighbors’ latest embeddings arrive.The vertex-level approach does not change aggregation results because it uses no embeddings or gradients from previous epochs.Source: §7.2.2 · p. 28

8 DISTRIBUTED GNN TRAINING SYSTEMS

The survey reviews 28 distributed GNN training systems and frameworks, grouping them chiefly by their computing resources. It summarizes their techniques using the survey’s taxonomy.

8 DISTRIBUTED GNN TRAINING SYSTEMS

  • The survey reviews 28 systems and frameworks, grouped into multi-GPU single machines, GPU clusters, CPU clusters, and miscellaneous resource settings.The table summarizes incorporated techniques according to the survey taxonomy; conventional, one-shot, and synchronous execution models are omitted as common basic models.Source: §8 · p. 28, Table 4 · p. 29

9 FUTURE DIRECTION

The survey identifies emerging directions for distributed GNN training, including benchmarking, dynamic and heterogeneous graphs, and model compression.

Benchmark for distributed GNN training

  • Few efforts compare distributed GNN training algorithms, and existing GNNMark profiling lacks flexibility and substantial attention to distributed, multi-machine scenarios.The survey calls for a benchmark designed for large-scale distributed GNN training.Source: §9 · p. 29

Large-scale dynamic graph neural networks

  • Dynamic graphs require partitioning that adapts to changing vertices and edges while balancing load and reducing communication; structural updates also lower cache hit ratios.The survey notes that reduced cache hit ratios significantly influence end-to-end training performance.Source: §9 · pp. 29–30

Large-scale heterogeneous graph neural networks

  • Heterogeneous graphs can cause load imbalance and high network overhead because feature sizes and neighbor counts vary by vertex type.The survey identifies distributed heterogeneous GNN support as an area needing more attention.Source: §9 · p. 30

GNN model compression technique

  • Distributed GNN compression remains underexplored; EC-Graph uses lossy message compression with compensation methods and bit tuning to balance accuracy and message size.The survey describes compression as orthogonal to other distributed GNN optimization techniques.Source: §9 · p. 30

10 CONCLUSION

The conclusion organizes distributed GNN training across its end-to-end pipeline, summarizes systems for three computing settings, and identifies future optimization directions.

10 CONCLUSION

  • The survey divides training into data partition, batch generation, and model training, then organizes optimization techniques into four orthogonal categories.The categories are GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol.Source: §10 · p. 30
  • The taxonomy covers partitioning, fast mini-batch generation, execution models for mini-batch and full-graph training, and synchronous and asynchronous communication protocols.Source: §10 · pp. 30–31
  • The survey summarizes representative systems for multi-GPUs, GPU clusters, and CPU clusters, and discusses future directions for distributed GNN training optimization.Source: §10 · p. 31

A DISTRIBUTED GNN TRAINING SYSTEMS

This section introduces representative distributed GNN systems and summarizes their evolution and efficiency techniques.

A DISTRIBUTED GNN TRAINING SYSTEMS

A.1 Systems on single machine with multi-GPUs

Single-machine multi-GPU systems optimize distributed GNN training through caching, partitioning, scheduling, data placement, and sampling pipelines.

A.1 Systems on single machine with multi-GPUs

  • NeuGraph uses locality-aware 2D graph partitioning, chunk scheduling, selective vertex transfer, and kernel optimizations for scalable full-graph training.Source: §A.1 · p. 39
  • PaGraph reports up to 96.8% less data transfer and up to 16× performance improvement over DGL, while BGL combines two-level caching with profiling-based resource allocation.Source: §A.1 · p. 40
  • GNNLab reports up to 9.1× and 74.3× performance over DGL and PyG, using pre-sampling caches and dynamic GPU allocation across executors.Source: §A.1 · p. 40
  • Rethinking models CPU–GPU data-movement costs to place data and uses locality-aware neighbor sampling to improve data-loading efficiency over DGL and PaGraph.Source: §A.1 · p. 40
  • DSP sends remote sampling tasks and returns sampled results rather than full neighbor lists; it reports up to 20× sampling speedup and over 2× lower overall training time in most cases.Source: §A.1 · pp. 40–41
  • Legion partitions around NVLink cliques and shares clique-local caches for graph topology and vertex features, supporting billion-scale graphs with significant end-to-end speedup.Source: §A.1 · p. 41

A.2 Systems on GPU clusters

GPU-cluster systems use partitioning, caching, hybrid parallelism, and pipelining to address communication and workload costs. Their designs range from full-graph training to mini-batch and decentralized approaches.

A.2 Systems on GPU clusters

  • P3’s push-pull parallelism partitions first-layer features by columns and uses model parallelism there, then data parallelism in later layers.The first layer transfers graph structure rather than vertex features; P3 also uses mini-batch pipelining and bounded-staleness asynchronous communication.Source: §A.2 · pp. 41–42
  • SANCUS uses historical embeddings and staleness-aware skip-broadcast to reduce communications by up to 74% without losing accuracy.It is decentralized, avoiding the centralized bottleneck bandwidth of parameter-server architectures.Source: §A.2 · p. 42
  • NeutronStar combines cached and communicated dependencies using a cost model and greedy heuristic, outperforming DistDGL by up to 1.8× and ROC by up to 14.3%.It decouples dependency management from graph operations and neural-network functions.Source: §A.2 · p. 42
  • DGCL plans communication using physical topology and GNN communication relations, reducing peer-to-peer communication time by 77.5% on average and epoch training time by up to 47%.Its plan divides transfers into link-based stages and selects a plan with a heuristic cost model and tree-spanning algorithm.Source: §A.2 · p. 42
  • PipeGCN’s one-epoch staleness bound reduces end-to-end training time by around 50% without compromising accuracy.It pipelines with historical remote embeddings and gradients while computing latest embeddings for local neighbors, and smooths boundary information with a moving average.Source: §A.2 · pp. 42–43
  • BNS-GCN communicates a stochastic subset of boundary vertices, combining full-graph accuracy with sampling-based training speed; DistDGLv2 reports 2–3× total training speedup over DistDGL.DistDGLv2 adds heterogeneous-graph partitioning and a parallel CPU mini-batch pipeline with hybrid CPU/GPU sampling.Source: §A.2 · p. 43
  • SALIENT++ combines probability-guided caching with a 10-stage feature-collection pipeline and reports 12.7× speedup over DistDGL [164].Its propagation model estimates each vertex’s sampling probability to select cache contents for each worker.Source: §A.2 · p. 43
  • G3 balances vertex, edge, and remote-neighbor workloads and overlaps communication with computation; AdaQP’s message quantization improves throughput by up to 3.02× with negligible accuracy drop.G3’s fine-grained pipeline overlaps work within and across layers; AdaQP adaptively assigns quantization bit widths and overlaps central-node computation with marginal-node communication.Source: §A.2 · pp. 43–44

A.3 Systems on CPU clusters

CPU-cluster systems support industrial-scale and distributed GNN workloads through sampling, graph partitioning, operator scheduling, and flexible execution abstractions. Their reported designs address communication, resource use, and model expressiveness in different ways.

A.3 Systems on CPU clusters

  • AliGraph organizes its industrial distributed GNN platform into application, algorithm, storage, sampling, and operator layers.Its sampling and operator abstractions allow users to select sampling methods and implement their own operators.Source: §A.3 · p. 44
  • AGL combines distributed k-hop batch generation, parameter-server training, and MapReduce inference, achieving nearly linear training speedup with 100 workers.It reports completing a 2-layer GAT training on billions of vertices in acceptable hours.Source: §A.3 · p. 44
  • DistDGL provides flexible distributed samplers and a distributed graph-and-feature KV store, with synchronous dense-parameter updates and asynchronous sparse-embedding updates.It is an open-source CPU-cluster framework for mini-batch GNN training.Source: §A.3 · p. 45
  • CM-GCN forms cohesive mini-batches that share neighbors to reduce remote-neighbor retrieval, while allowing vertex-level asynchronous computation.The authors note possible training bias relative to the original GCN, but report accuracy similar to GCN for CM-GCN and BNS-GCN.Source: §A.3 · p. 45
  • DistGNN offers bias, synchronous, and bounded-staleness asynchronous aggregation for vertex-cut partitions, with the asynchronous mode reducing communication costs.It uses Libra partitioning for balanced workloads and a Type I asynchronous execution model.Source: §A.3 · p. 45
  • FlexGraph’s NeighborSelection, Aggregation and Update abstraction supports models with indirect neighbors during aggregation, which GAS-like abstractions cannot express.It uses hierarchical dependency graphs and hybrid aggregation strategies, and is reported to gain expression and scaling capability over DGL, PyG, NeuGraph, and Euler.Source: §A.3 · p. 45
  • ByteGNN targets CPU-cluster resource utilization with a sampling abstraction and operator-parallel DAG scheduling, improving CPU utilization and achieving up to 3.5× speedup over DistDGL.Its two-level scheduler schedules DAGs and their inner operators; its graph partitioning also addresses workload imbalance and network overhead.Source: §A.3 · pp. 45–46
  • ParallelGCN sends only necessary vertex embeddings with non-blocking peer-to-peer communication and reports up to 41% communication reduction using hypergraph partitioning.Its chunk-based execution overlaps computation on received chunks with communication of other chunks.Source: §A.3 · p. 46

A.4 Miscellaneous

The miscellaneous systems explore serverless computation and resource-constrained devices as alternatives for GNN training.

A.4 Miscellaneous

  • Dorylus assigns graph operations to CPU clusters and dense neural operations to serverless threads, using pipelining and bounded-staleness asynchronous communication.The paper reports serverless-thread efficiency close to GPUs at lower cost, while its asynchronous pipeline reaches higher accuracy than sampling-based GNN training methods.Source: §A.4 · p. 46
  • SUGAR trains local GNN models on expanded subgraphs across limited-resource devices, maintaining K local models rather than a global model to save computation, memory, and communication.It provides complexity, error-bound, convergence, and graph-partition proofs, and reports considerable runtime and memory usage compared with state-of-the-art GNN models on large-scale graphs.Source: §A.4 · p. 46

Related papers

Loading 2211.00216v2…