PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Augmentation for Graph Neural Networks

Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, Dinghao Wu

arXiv:2109.03856v4cs.LG

TL;DR

GNNs aggregate local neighborhoods, but whether this provides enough information for nodes with few neighbors remains an open question. Local augmentation learns neighbor-feature distributions conditioned on a central node’s feature and supplies generated features to GNNs; experiments report gains across architectures and benchmarks.

  • Problem

    Whether neighborhood information is adequately aggregated for effective representations of nodes with few neighbors remains an open question.

    Source: §1 · p. 1

  • Method

    Local augmentation learns the conditional distribution of neighbors’ features given a central node’s feature and supplies generated features as additional input to GNNs.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    3.4% and 1.6% average test-accuracy gains over GCN and GAT, respectively, on Cora, Citeseer, and Pubmed accompany reported improvements across GNN architectures and benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that local augmentation improves performance across GNN architectures and benchmark datasets, with reported state-of-the-art results on semi-supervised node classification tasks.

    Source: §6 · p. 9

    The method assumes a neighbor’s feature distribution is independent of its non-descendants given the central node, without modeling other nodes connected to that neighbor.

    Source: §3.1 · pp. 3–4

Abstract

from arXiv · show

Graph Neural Networks (GNNs) have achieved remarkable performance on graph-based tasks. The key idea for GNNs is to obtain informative representation through aggregating information from local neighborhoods. However, it remains an open question whether the neighborhood information is adequately aggregated for learning representations of nodes with few neighbors. To address this, we propose a simple and efficient data augmentation strategy, local augmentation, to learn the distribution of the node features of the neighbors conditioned on the central node's feature and enhance GNN's expressive power with generated features. Local augmentation is a general framework that can be applied to any GNN model in a plug-and-play manner. It samples feature vectors associated with each node from the learned conditional distribution as additional input for the backbone model at each training iteration. Extensive experiments and analyses show that local augmentation consistently yields performance improvement when applied to various GNN architectures across a diverse set of benchmarks. For example, experiments show that plugging in local augmentation to GCN and GAT improves by an average of 3.4\% and 1.6\% in terms of test accuracy on Cora, Citeseer, and Pubmed. Besides, our experimental results on large graphs (OGB) show that our model consistently improves performance over backbones. Code is available at https://github.com/SongtaoLiu0823/LAGNN.

1. Introduction

The introduction identifies sparse neighborhoods as a challenge for node representations and proposes conditional generation of neighborhood features as a flexible augmentation for GNNs. It reports improvements across citation networks and large-scale OGB datasets.

1. Introduction

  • GNNs use message passing to aggregate local-neighborhood information, while prior work also emphasizes locality through deep architectures and neighborhood-aware aggregation schemes.Existing graph augmentation approaches instead perturb topology or node features from a global perspective, neglecting neighborhood-local information.Source: §1 · pp. 1–2
  • Limited neighbors may restrict expressive power and hinder representations, especially for low-degree nodes; simply expanding receptive fields can cause over-smoothing.The introduction identifies enriching local information for low-degree nodes as the target problem.Source: §1 · p. 1
  • LA-GNN learns neighbor-feature distributions conditioned on a central node’s feature, then supplies sampled features as additional input during GNN training.Decoupled generative-model pretraining makes the augmentation applicable to different GNN backbones.Source: §1 · p. 2
  • 3.4% and 1.6% average test-accuracy gains over GCN and GAT, respectively, on Cora, Citeseer, and Pubmed accompany superior LAGNN performance on large-scale OGB datasets.The reported gains are averages over the three citation networks.Source: §1 · p. 2
  • The paper presents local-neighborhood feature generation as a general augmentation strategy and graph generative-model pretraining as a direction for improving downstream task performance.The framework is described as flexible across popular backbones and benchmarks.Source: §1 · p. 2

2. Preliminaries

The preliminaries define the graph and feature notation, then describe GNN layers as repeatedly aggregating immediate-neighbor representations and combining them with the node’s representation.

2. Preliminaries

  • The graph is G = (V, E), with vertices V, edges E, adjacency matrix A, neighborhood N_i, and diagonal degree matrix D.The node feature matrix is X ∈ R^{N×F}, and Y denotes the one-hot label matrix.Source: §2 · p. 2
  • GNNs operate on graph structure by repeatedly aggregating immediate-neighbor representations and combining the result with the node representation.The resulting representation is denoted h_v.Source: §2 · p. 2
  • A message-passing layer uses AGGREGATE and COMBINE functions; h_v^(k) denotes node v’s representation at layer k, and e_{u,v} denotes the edge vector.The displayed scheme names the aggregation and combination operations.Source: §2 · p. 2

3. Local Augmentation for Graph Neural Networks (LAGNN)

Local augmentation learns a conditional distribution of neighbor features and supplies generated features alongside original features to downstream GNNs. Its generative-model pretraining is decoupled from GNN training so the augmentation can be used with different backbones.

3.1. Local Augmentation

  • A CVAE learns neighbor-feature distributions conditioned on the center node’s feature, then samples generated features for use alongside the original features.It is trained on neighboring feature pairs to maximize the ELBO; at generation time, the center feature conditions the model and a latent variable is sampled.Source: §3.1 · p. 3
  • The neighbor-feature model assumes those features are independent of their non-descendants given the center node, avoiding costly multi-hop conditioning.The authors say this assumption improves scalability while reporting strong performance across benchmarks.Source: §3.1 · pp. 3–4

3.2. Decoupling the Generative Model Training from Downstream Graph Learning

  • The method factors the GNN and generative distributions, enabling one-time unsupervised pretraining before applying the augmentation to different GNN models.The generated feature matrix is incorporated as additional input to the GNN.Source: §3.2 · p. 4

3.3. Architecture

  • The architecture offers concatenation and averaging designs; concatenation modifies the first layer while preserving its parameter size, including for GAT.For GAT, the concatenation design replaces the input to half of the attention heads with generated features.Source: §3.3 · p. 4

3.4. Loss Functions

  • Training uses supervised loss on original and generated features, with optional consistency regularization to encourage invariant predictions across inputs.The consistency loss is described as optional and intended for specific GNNs and graph-learning tasks.Source: §3.4 · p. 5
  • At each training iteration, the method samples generated features and uses a different sample for validation; inference selects features by validation loss or accuracy.The authors report that generative-model pretraining usually takes fewer than 10 epochs and introduces little computational overhead.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate local augmentation across citation-network and large-graph tasks, with additional studies of its components, missing-feature robustness, node-degree behavior, and over-smoothing.

4.1. Semi-supervised Learning

  • Experiments on Cora, Citeseer, and Pubmed use standard fixed splits and evaluate concatenation-style LAGNNs built on GCN, GAT, and GCNII against other augmentation models and subgraph GNNs.The splits use 20 training nodes per class, 500 validation nodes, and 1,000 test nodes.Source: §4.1 · p. 6
  • GCN gains 3.1%, 4.4%, and 2.7% on Cora, Citeseer, and Pubmed; GAT gains 1.7%, 1.2%, and 2.0%, respectively.The reported mean classification accuracy is averaged over 100 runs.Source: §4.1 · p. 6, Table 1 · p. 6

4.2. Full-supervised Learning

  • On full-supervised OGB node and link prediction, local augmentation consistently improves performance over the backbones.The evaluation includes node-property prediction and link prediction.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.4. Ablation Study

  • On Pubmed, adding generated features without consistency training improves GCN test accuracy by 1.8%, while the architecture-only change improves it by 0.3%.Consistency training and sharpening further enhance performance.Source: §4.4 · pp. 7–8, Table 5 · p. 8

4.5. Robustness to Missing Information

  • As feature-mask ratios rise, the GCN–LAGCN performance gap widens in most Citeseer cases.The study interprets this as evidence that local augmentation can complement local-neighborhood context.Source: §4.5 · p. 8, Table 6 · p. 8

4.6. Case Study

  • On Pubmed, about 76.1% of test nodes have degree below 6, and lower-degree nodes tend to have lower test accuracy.The authors report that local augmentation enriches local information for these nodes and enhances their performance.Source: §4.6 · p. 8, Table 7 · p. 8

4.7. Over-smoothing Analysis

  • LAGCN’s MADgap is at least as large as GCN’s across the reported Cora layer depths; the authors say this can alleviate over-smoothing.They state that addressing over-smoothing is not the approach’s objective.Source: §4.7 · p. 8, Table 8 · p. 8

5. Related Work

Related work spans unsupervised graph representation learning, graph generative models, and feature concatenation designs. The paper distinguishes its approach as generating neighborhood features for downstream backbones.

Unsupervised Representation Learning on Graphs

  • Unsupervised graph representation learning includes contrastive self-supervision, graph embeddings, and random walks; the paper also characterizes local augmentation as unsupervised.Contrastive methods minimize distances for positive pairs and maximize distances for negative pairs; random-walk methods learn node representations from walk-derived sequences.Source: §5 · pp. 8–9

Graph Generative Models

  • Unlike graph generative approaches focused mainly on structure generation, this model generates feature representations to enhance downstream backbone models.The related work discusses VGAE, generative GCNs, and conditional structure generation, among other graph-generation approaches.Source: §5 · p. 9

Concatenation-style Design

  • The concatenation-style design joins original features with different generated features at each training iteration to enrich neighborhood information.The paper notes that concatenation is also used by GAT and SIGN; SIGN augments features with multi-hop information through powers of adjacency.Source: §5 · p. 9

6. Conclusion

Local augmentation learns a generative distribution of neighbor features conditioned on the central node’s feature and feeds generated features to modified GNN backbones. Experiments report improvements across architectures and benchmarks, while the framework is limited to the central node’s immediate neighbors.

6. Conclusion

  • The method learns the conditional distribution of neighbors’ features given a central node’s feature, then supplies generated features to modified backbone GNNs.The paper describes local augmentation as a general feature-generation strategy for improving existing GNNs.Source: §6 · p. 9
  • Experiments report improved performance across GNN architectures and benchmark datasets, with new state-of-the-art results on semi-supervised node-classification tasks.The framework does not exploit 2-hop neighbors or use random walks to identify additional related neighbors.Source: §6 · p. 9

A. Proofs

The appendix provides additional material on the generator’s ELBO, CVAE framework, active-learning pre-training, and how local augmentation differs from reconstruction-based graph embedding methods.

A. Proofs

  • The paper provides further derivation details for the generator ELBO in Eq. (2).The supplied passage introduces the derivation but does not give its steps.Source: A. Proofs · p. 13

B.1. Framework

  • The CVAE uses central-node and neighbor feature vectors as inputs; at inference, it conditions the decoder on the central feature and a sampled latent variable.Figure 2 depicts the central node and its neighbors as CVAE inputs.Source: §B.1 · p. 13, Figure 2 · p. 14

B.2. Active Learning Trick

  • For Cora, Citeseer, and Pubmed, pre-training uses BALD with MC-dropout samples to prioritize generated feature matrices with greater uncertainty.A high BALD score indicates greater network uncertainty about the generated feature matrix.Source: §B.2 · p. 14

C.1. Connection to EP-B and GraphSAGE

  • Unlike EP-B and GraphSAGE, local augmentation does not assume nearby nodes share similar attributes; it generates connected-neighbor features from a learned distribution.The figure contrasts reconstruction of a central embedding, similarity between nearby embeddings, and inference of neighbor representations from the central representation.Source: §C.1 · p. 15, Figure 3 · p. 15

D.1. Datasets Details

The evaluation covers citation networks and six OGB datasets spanning node classification, link prediction, and graph prediction; a figure also contrasts two LAGCN architectures.

D.1. Datasets Details

  • Cora, Citeseer, and Pubmed are citation networks whose nodes are documents, edges are citations, features are bag-of-words, and labels are academic topics.The paper also evaluates on six large OGB datasets.Source: §D.1 · p. 15
  • The six OGB datasets cover node classification, link prediction, and graph prediction across product, protein, paper, collaboration, and molecular graphs.The node-classification datasets are ogbn-arxiv, ogbn-proteins, and ogbn-products; ogbl-collab is for link prediction, and ogbg-molhiv and ogbg-molpcba are for graph prediction.Source: §D.1 · p. 15
  • The concatenation-style LAGCN adds a convolutional layer for X and combines hidden representations by concatenation, unlike the average-style architecture.Source: Figure 4 · p. 16

D.2. Implementation Details

The implementation uses established frameworks and reference implementations for LA-GNN variants, with experiments run on a single NVIDIA Tesla V100.

D.2. Implementation Details

  • The implementations use PyTorch, PyG, and DGL, adapting reference implementations for the corresponding GCN, GAT, GCNII, GRAND, and OGB models.Source: §D.2 · p. 16
  • All experiments ran on one NVIDIA Tesla V100 with 32GB of memory.Source: §D.2 · p. 16
  • The reported software stack includes Python 3.6.8, PyTorch 1.9.0, PyG 2.0.3, OGB 1.3.2, and DGL 0.7.2.The passage also lists CUDA, cuDNN, NumPy, torchvision, and PyTorch extension versions.Source: §D.2 · p. 16

D.3. Hyperparameter Details

The appendix describes two LA-GNN design styles and directs readers to hyperparameter tables for citation and OGB datasets.

D.3. Hyperparameter Details

  • Concatenation-style LA-GNNs add an aggregation over generated feature matrix X before concatenation.The passage also identifies average and concatenation as the two design styles.Source: §D.3 · p. 16
  • For Pubmed, the first layer uses 3 additional generated feature matrices.Source: §D.3 · p. 16
  • Tables 10 and 11 list backbone hyperparameters for three citation datasets and OGB datasets, respectively.Source: §D.3 · p. 16, Table 10 · p. 17, Table 11 · p. 18

Related papers

Loading 2109.03856v4…