PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training

Chuxiong Sun, Hongming Gu, Jie Hu

arXiv:2104.09376v3cs.LGcs.AI

TL;DR

Large graphs are difficult to handle with conventional GNNs, and scalable models have limited study of combining self-training with label propagation. The paper proposes SAGN’s hop-attention classifier and the SLE training framework; experiments report competitive results across inductive and transductive settings, with SLE further improving performance.

  • Problem

    Conventional GNNs are difficult to scale to large graphs, and combining self-training with label propagation in scalable GNNs is underexplored.

    Source: §1 · pp. 1–2

  • Method

    SAGN uses learnable attention across neighborhood hops, while SLE combines a scalable label model, label propagation, and staged self-training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    SAGN+SLE outperforms sampling-based and sampling-free baselines on most datasets, and SLE improves SAGN by up to 8.4% relative.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The proposed sampling-free methods are described as easy to implement and extend to huge graphs with competitive performance.

    Source: §5 · p. 9

    The multi-hop input requires CPU memory proportional to the number of hops, and SLE brings less improvement on some inductive datasets.

    Source: §5 · p. 9

Abstract

from arXiv · show

It is hard to directly implement Graph Neural Networks (GNNs) on large scaled graphs. Besides of existed neighbor sampling techniques, scalable methods decoupling graph convolutions and other learnable transformations into preprocessing and post classifier allow normal minibatch training. By replacing redundant concatenation operation with attention mechanism in SIGN, we propose Scalable and Adaptive Graph Neural Networks (SAGN). SAGN can adaptively gather neighborhood information among different hops. To further improve scalable models on semi-supervised learning tasks, we propose Self-Label-Enhance (SLE) framework combining self-training approach and label propagation in depth. We add base model with a scalable node label module. Then we iteratively train models and enhance train set in several stages. To generate input of node label module, we directly apply label propagation based on one-hot encoded label vectors without inner random masking. We find out that empirically the label leakage has been effectively alleviated after graph convolutions. The hard pseudo labels in enhanced train set participate in label propagation with true labels. Experiments on both inductive and transductive datasets demonstrate that, compared with other sampling-based and sampling-free methods, SAGN achieves better or comparable results and SLE can further improve performance.

1 Introduction

Scalable GNNs simplify minibatch training on large graphs but may lose expressiveness, while combining self-training and label propagation in scalable models remains underexplored. The paper introduces SAGN and SLE and reports competitive results across inductive and transductive datasets.

Motivation

  • Scalable GNNs decouple graph convolutions from learnable transformations for efficient minibatch training, but preprocessing convolutions can limit expressiveness.The paper argues that comparable performance requires a more expressive, graph-structure-aware classifier.Source: §1 · pp. 1–2
  • Prior work applies self-training and label propagation to common GNNs, but their combination in scalable GNNs is underexplored and per-epoch masking-based propagation is costly.The introduction identifies runtime cost from repeating label propagation with inner random masking each epoch.Source: §1 · p. 2

Contributions

  • SAGN replaces SIGN’s redundant concatenation with learnable attention across multiple hops to improve expressiveness and interpretability.Its attention weights adaptively combine neighborhood information across hops.Source: §1 · p. 2
  • SLE combines self-training and label propagation for scalable GNNs, using a scalable label model and staged training-set enhancement.The authors report that enough propagation empirically reduces label leakage without inner random masking, and hard pseudo labels join true labels in propagation.Source: §1 · p. 2

Experiments

  • SAGN+SLE achieves state-of-the-art results on ogbn-products and ogbn-papers100M, while SLE improves SAGN by up to 8.4% relative.Across inductive and transductive datasets, SAGN also outperforms sampling-based and sampling-free baselines on most datasets.Source: §1 · p. 2

2 Background and related works

The related work contrasts sampling-based scaling with preprocessing-based scalable GNNs, then reviews self-training and label propagation as routes to improve semi-supervised graph learning.

Sampling-based Methods

  • Sampling-based methods include node-wise, layer-wise, and graph-wise approaches, but can require complex implementation and retain redundant convolution computation.Common GNNs also face scaling difficulty because sampled-node receptive fields vary and grow exponentially with layers.Source: §2 · p. 2

Scalable GNNs

  • Scalable GNNs decouple preprocessing from a classifier; SGC is faster but its single linear layer limits expressiveness and loses multi-hop information.SGC removes intermediate nonlinear activations and achieves comparable performance on small datasets.Source: §2 · pp. 2–3
  • SIGN uses hop-specific MLP encoders and concatenation to approximate nonlinear hierarchies, but uniform concatenation ignores node-specific hop importance and costs extra memory.SIGN substantially outperforms SGC.Source: §2 · p. 3

Self-training approach on graphs

  • Graph self-training adds pseudo-labeled nodes, while prior approaches may require multiple models, topology updates, or high-quality predictions and clustering.Self-Enhanced GNN can add redundant edges or remove valuable ones when predictions are low quality.Source: §2 · p. 3

Label propagation

  • Label propagation methods incorporate labels through concatenated or transformed embeddings, edge-weight priors, or propagated residuals and corrections.Correct and Smooth is scalable postprocessing, though simple base models can require costly spectral and diffusion preprocessing.Source: §2 · p. 3

3 Proposed methods

SAGN adapts scalable GNNs by encoding multi-hop features and combining them with attention. SLE adds a scalable label model and staged self-training, using propagated true and hard pseudo labels.

3.1 SAGN

  • SAGN follows scalable-model architecture and incorporates attention to integrate information from multiple neighborhood hops.It uses a residual term and a post-MLP encoder to produce its output representation.Source: §3.1 · p. 4
  • A non-parametric transition matrix supports preprocessing of neighborhood features across the selected hops.The matrix may be a row-stochastic random walk matrix or a symmetrically normalized adjacency matrix.Source: §3.1 · p. 4
  • SAGN encodes each hop's node features with an MLP, then sums the encoded representations using diagonal attention matrices.The paper hypothesizes that the useful multi-hop information varies among nodes.Source: §3.1 · p. 4

3.2 SLE

  • SLE combines a base model with a scalable label model, removing inner random masking and allowing high-confidence hard pseudo labels to join true labels in propagation.The label model receives propagated label vectors, and its output is combined with the base-model output.Source: §3.2 · pp. 4–5, Figure 3 · p. 5, Figure 4 · p. 5
  • SLE trains in stages: it starts with the raw training set, then uses prior predicted probabilities to enhance the set before training each subsequent model.Its stage-specific loss is optimized with Adam; the paper reports not using knowledge-distillation loss because it conflicts with hard pseudo labels and empirically brings no improvements.Source: §3.2 · pp. 5–6

4 Experiments and analysis

Experiments span inductive and transductive datasets, where SAGN and SLE are evaluated against scalable and sampling-based baselines. Results show strong transductive performance, while inductive gains from SLE vary by dataset; runtime and ablation analyses examine efficiency and component contributions.

Experimental settings

  • Evaluation covers inductive Reddit, Flickr, PPI, and Yelp and transductive ogbn-products, ogbn-papers100M, and ogbn-mag; inductive training restricts graph convolutions for raw training nodes to raw train graphs.In the inductive setting, nodes outside the training set are excluded from training but allowed at inference.Source: §4 · p. 6

Inductive results

  • SAGN beats SIGN by 0.9% on PPI and 2.2% on Yelp; SLE improves Flickr by 3.2% but brings few gains on most inductive datasets.The label model is omitted at all stages for Yelp because it empirically brings no improvements.Source: Table 2 · p. 7, §4 · p. 7

Transductive results

  • On ogbn-products and ogbn-papers100M, SAGN+2-SLE achieves state-of-the-art results, improving over SAGN by 3.08% and 1.25%, respectively.The reported improvements are based on SAGN.Source: §4 · p. 7, Table 3 · p. 8
  • On ogbn-products SAGN beats SIGN by 0.68%; on ogbn-papers100M it beats methods without labels by at least 1.07%.SAGN+2-SLE also outperforms UniMP on ogbn-papers100M, including at the first stage.Source: §4 · p. 7, Table 3 · p. 8

Complexity

  • SAGN and SIGN share total time complexity KLNd2, while SAGN has lower post-encoder complexity and negligible attention overhead.The analysis assumes multi-hop and post-MLP encoder layer counts have the same order of magnitude L.Source: §4 · p. 7, Table 4 · p. 8

Runtime

  • SAGN and SIGN have very close training and inference runtimes; SAGN uses 10% less memory and 36% fewer parameters.The savings are attributed to removing concatenation; adding the label model increases training time and memory, while inference runtime remains stable.Source: Table 5 · p. 8, §4 · p. 8

Ablation studies

  • On ogbn-products, each base model with SLE beats its SE counterpart by at least 1.81%; combining MLP and the label model yields at least 3.56% improvement.The combined model can gain up to 1.49% more with the enhanced training set; the authors hypothesize MLP captures less overlapping information with the label model.Source: Table 6 · p. 9, §4 · p. 9

5 Conclusion

The conclusion presents SAGN and SLE as scalable approaches for semi-supervised learning on large graphs. The accompanying appendices explain the attention-based replacement for SIGN’s concatenation and how SLE uses hard pseudo-labels.

5 Conclusion

  • SAGN+SLE outperforms sampling-based and sampling-free baselines on most datasets, while SAGN has complexity and runtime similar to SIGN.The authors also report that SLE can boost MLP, SIGN, and SAGN.Source: §5 · p. 9
  • The authors identify limited transition-matrix choices and hop-proportional CPU memory as constraints; SLE brings less improvement on some inductive datasets.They also report longer training runtime with expanded training sets, alongside faster convergence.Source: §5 · p. 9

A.1 Concatenation operation in SIGN

  • Learnable attention weights replace SIGN’s unweighted concatenation, improving expressiveness while reducing memory cost at additional complexity.The appendix notes that concatenation followed by a linear layer is equivalent to applying multiple linear layers and summing them.Source: §A.1 · p. 13

A.2 SLE

  • At each SLE stage, the model trains on an expanded labeled set and recomputes label propagation using updated initial label embeddings.The process adds newly included training nodes and updated hard pseudo-labels.Source: §A.2 · p. 13

A.3 Hard pseudo labels

  • SLE filters confident nodes for hard pseudo-labeling, then propagates those labels with true labels to extend teacher-model knowledge.The text names threshold-based and top-K filtering as examples.Source: §A.3 · p. 13

B.1 Dataset details

The appendix describes inductive and transductive node-classification datasets, their splits and features, and the baseline and setup choices used in experiments.

B.1 Dataset details

  • Reddit is an inductive post-community classification task; training uses the first 20 days of September 2014 and testing the remainder.Nodes represent posts, edges connect posts commented on by the same user, and features are 300-dimensional GloVe vectors.Source: §B.1 · p. 13
  • Flickr predicts picture tags from low-level features, while Yelp predicts users’ visited-business types using review-derived features; both are inductive tasks.Flickr uses a 0.5/0.25/0.25 train-validation-test split; Yelp uses 0.75/0.10/0.15.Source: §B.1 · pp. 13–14
  • PPI predicts protein roles from interaction subgraphs using gene-set, motif, and immunological-signature features; 20 graphs train and two each validate and test.The passage describes the train, validation, and test graphs as separate.Source: §B.1 · pp. 13–14
  • ogbn-products is transductive product-category prediction, split by sales rank: the top 8% train, next 2% validate, and remainder test.Product nodes have 100-dimensional bag-of-words description features.Source: §B.1 · p. 14
  • ogbn-papers100M predicts arXiv subject areas with a time-based split, while ogbn-mag predicts paper venues on a heterogeneous graph with several node and relation types.The papers100M split uses papers through 2017 for training, 2018 for validation, and papers since 2019 for testing.Source: §B.1 · pp. 14–15

B.2 Inductive setting

  • For inductive experiments, feature and label propagation is restricted to subgraphs, and confident raw-training nodes are also filtered when initializing label embeddings.Other zero-initialized raw-training nodes still participate in model training.Source: §B.2 · p. 15

B.3 Baseline selection

  • Baselines span vanilla, sampling-based, and sampling-free methods for inductive datasets, with dataset-specific scalable and label-informed methods on the transductive benchmarks.The listed comparisons include SGC, SIGN, UniMP, MLP+C&S, NARS, and several sampling-based GNNs.Source: §B.3 · p. 15

B.4 Setup

  • For ogbn-mag, non-paper nodes without raw features receive either averaged neighboring paper attributes or pretrained TransE embeddings.The TransE embeddings also incorporate heterogeneous information.Source: §B.4 · p. 15

C.1 Experiments on ogbn-mag

On ogbn-mag, SAGN with SLE improves results in homogeneous and heterogeneous settings. NARS_SAGN+2-SLE reaches the reported state-of-the-art mean test accuracy.

C.1 Experiments on ogbn-mag

  • 54.40% mean test accuracy: NARS_SAGN+2-SLE is reported as state of the art on ogbn-mag.The experiment uses 256-dimensional TransE embeddings and example relation subsets from NARS; Table 7 reports validation and test accuracy means and standard deviations.Source: §C.1 · p. 16, Table 7 · p. 16
  • 42.75% mean test accuracy: SAGN with an enhanced training set improves over its first-stage result in the homogeneous setting.SAGN at the first stage slightly outperforms SIGN; SIGN reaches 40.46% mean test accuracy.Source: §C.1 · p. 16, Table 7 · p. 16
  • 50.29% mean test accuracy: SAGN with increasing SLE stages improves in the heterogeneous setting using TransE embeddings.The passage says this exceeds most GNNs designed for heterogeneous graphs except NARS.Source: §C.1 · p. 16, Table 7 · p. 16
  • NARS+SLE combines relation-subgraph feature matrices with a weighted aggregator, then feeds the combined hop features into SAGN or SIGN.The figure identifies each relation subset’s induced subgraph and the whole graph as inputs.Source: Figure 5 · p. 17

C.2 Runtime experiments

Runtime experiments report stage-dependent training cost on ogbn-products and compare SIGN with SAGN on ogbn-papers100M. SAGN has lower memory and parameter use, with slightly longer inference time.

C.2 Runtime experiments

  • Training runtime increases with SLE stage on ogbn-products as the enhanced training set grows, while the model converges in fewer epochs.The table caption likewise describes more training time but convergence in fewer epochs.Source: §C.2 · p. 17, Table 8 · p. 17
  • On ogbn-papers100M, SIGN and SAGN have close results; SAGN uses less GPU memory and fewer parameters but has slightly longer inference runtime.The passage gives no specific numerical difference for this comparison.Source: §C.2 · p. 17

D Converge analysis

Enhanced training sets help models converge faster, while the effect of increasing stages on validation accuracy varies by dataset. The appendix also reports hyperparameter and runtime settings for the evaluated configurations.

D Converge analysis

  • Models with enhanced training sets converge faster; with a label model, all base models converge faster and show similar curves, while MLP without a label model gains no self-training improvement.Later-stage models can benefit from previous model outputs, allowing earlier training cutoffs to control time without losing performance.Source: D Converge analysis · p. 17, Figure 6 · p. 18, Figure 7 · p. 18

E Hyperparameter settings

  • Runtime experiments on ogbn-papers100M report mean and standard deviation over 10 runs, memory, and parameter counts; settings include batch sizes 5,000/100,000, hidden dimension 512, and three aggregations.On ogbn-products, runtime settings use batch sizes 50,000/100,000, hidden dimension 512, and five aggregations; SIGN+0-SLE encounters OOM.Source: Table 9 · p. 17, E Hyperparameter settings · p. 20

D Converge analysis

  • As stages increase, validation and test accuracy improve on ogbn-papers100M and ogbn-mag, but validation accuracy falls slightly on ogbn-products, where the validation ratio is 2%.The authors attribute closer validation and test accuracies and improved overall results to alleviating best-model-selection overfitting when the enhanced set includes validation and test nodes.Source: D Converge analysis · p. 18

E Hyperparameter settings

  • Hyperparameters are tuned empirically; SAGN+SLE omits the label model at stage one, and uses three hops rather than five in the listed ablation configuration.SIGN, SAGN, and the other listed variants share two MLP layers and hidden dimension 512.Source: Table 10 · p. 18, E Hyperparameter settings · p. 19

F Visualization

Attention-weight heatmaps show different hop-importance patterns across ogbn-products and ogbn-papers100M. On papers100M, label-model stages make hop importances more homogeneous, while node-to-node differences remain visible.

F Visualization

  • On ogbn-products, the 0-hop feature has larger average attention weight than other hops, though its weight can be small for some nodes; adding a label model or self-training does not change hop importances.The heatmap axes are hop number horizontally and node id vertically.Source: F Visualization · p. 20, Figure 12 · p. 22
  • On ogbn-papers100M, 2-hop and 3-hop features are more important, and adding the label model makes hop importances more homogeneous.The distribution shifts slightly at stage one and returns at stage two; weights also differ across nodes, visible as color blocks within a column.Source: F Visualization · p. 21, Figure 13 · p. 23
Loading 2104.09376v3…